A specialist model with a sensible job
Needle 2 is not trying to be a tiny replacement for a conversational model. It has one main job: read a request, select a declared function, and produce arguments that fit its schema. Structured extraction is treated as the same task with one available function. Many local assistants do not need essays or broad world knowledge. They need to turn "dim the living room to 30" into a safe, predictable command.
Cactus Compute says the 45-million-parameter model is compressed into a single 14 MB binary and uses about 28 MB of memory for a full session. The repository contains the Python interface, inference path, LoRA fine-tuning workflow, and exporter. A native engine and weights are fetched from Hugging Face on first use, cached, and then run without a network connection. For an appliance, wearable, smart-home controller, or robot, that size and offline behavior could matter more than fluent conversation.
The MIT license is friendly, although the Python package metadata currently says Apache 2.0 while the repository's actual license file says MIT. That inconsistency should be fixed.
Constrained output is the strongest idea
Needle compiles a byte-level grammar from each tool's JSON schema. The generated call therefore has to match declared types, enumerations, numeric ranges, string patterns, and array limits. A malformed JSON object or out-of-range value is blocked during decoding instead of being noticed after generation. Python decorators derive schemas from signatures and docstrings, while Field adds tighter constraints and Pydantic models provide typed extraction.
This is a meaningful improvement over parsing whatever a small model happens to write. It still does not prove that the selected tool or value is semantically correct. If the user asks for one recipient and the model chooses another valid recipient, grammar cannot save you. Applications must validate permissions, confirm high-impact actions, and test ambiguous language.
The confidence mechanism is designed for that uncertainty. Each answer includes a score based on a learned head and call-token probabilities. The README recommends choosing a threshold, acting above it, and escalating below it. Teams still need to calibrate that threshold on their own commands.
Large tool catalogs fit, long context does not
When five or fewer tools are declared, Needle renders all of them. With a larger catalog, a built-in retrieval head embeds the query and exposes only the five highest-scoring tools to the model. The decode grammar is rebuilt around that subset, so an unselected tool cannot be called. Embeddings can be persisted to avoid rebuilding an unchanged index. This gives a small model a route into a much larger action catalog without placing every schema in its prompt.
The cost is another failure boundary. A correct tool omitted by retrieval is unreachable, no matter how well the calling model understands the request. Test retrieval recall separately from argument accuracy. Similar tool names and overlapping descriptions deserve extra attention. The README says writing good tool descriptions is central to results, which puts real design work on the integrator.
Conversation memory is deliberately bounded. A 256-token sliding window limits memory growth, while tool definitions remain pinned as key-value sinks. That suits short command chains and device control, but it is not a substitute for a long support conversation or a complex planning agent. Important state should live in the application and be supplied as explicit facts or tool results.
Easy installation does not mean finished integration
The promised trial is short: install cactus-needle, decorate a Python function, create an agent, and run a request. The engine download happens once. Needle supports direct schemas, manual tool loops, a browser playground, and a helper for typed extraction. Python 3.9 or newer is declared, and platform-specific native libraries are selected for macOS, Windows, and Linux.
The package is heavier than the 14 MB model headline suggests. Its Python dependencies include JAX, JAXlib, Flax, Optax, NumPy, SentencePiece, and Hugging Face Hub. More importantly, issue 46, opened August 10, shows the exact weather quickstart returning a normal response with no function call on macOS. A second user reproduced it on Linux. That is a direct reason to test before designing around the API.
The mobile claim also needs careful reading. The model and Cactus engine target tiny devices, but this repository documents a Python package rather than end-to-end Android, iOS, or React Native integration. Mobile developers should evaluate the broader Cactus runtime and confirm application bindings.
Fine-tuning is accessible, but not trivial
Needle accepts JSONL examples pairing a query and tool schemas with expected calls. It can optionally synthesize data through OpenRouter, run LoRA fine-tuning, merge the adapter, quantize the result, and emit a custom .cact binary. A playground button wraps much of that workflow.
Good data remains the hard part. Synthetic examples need review, off-topic cases need explicit empty answers, and every sensitive action needs adversarial prompts. The intermediate workflow uses pickle checkpoint files. Issue 36 correctly warns that loading untrusted pickle data can execute code. The final runtime artifact is a different format, but training teams should still control checkpoint provenance and storage.
Young, active, and not yet settled
The repository was created in February 2026 and pushed on August 10. Recent merged work fixed JAX compatibility and false TPU detection, while open pull requests cover decoding speed and correctness. The GitHub count showed 28 open issues and pull requests, and the project had no GitHub release even though its package identifies itself as version 2.0.0.
Needle is ready for a focused evaluation, not blind adoption. Build a test set from real utterances, measure tool retrieval and arguments separately, set confidence thresholds from those results, and require confirmation for irreversible actions. If it clears that bar, very few alternatives offer this much local tool-calling machinery in such a small claimed footprint.