A 45M-parameter model handles narrow tool decisions locally
Needle is built for a smaller job than a general assistant. You provide Python functions or JSON schemas, and the model chooses a tool and fills its arguments under a byte-level grammar. It can also extract a Pydantic object from text. The model has 45 million parameters, while the published engine is described as a single 14 MB binary using about 28 MB of memory. Those are project claims, not figures from our lab run, but they explain the appeal: a device can route a command without sending it to a cloud model.
The Python API has two useful levels. run() executes decorated functions and feeds results back into the session. complete() leaves execution to the caller, which is the safer fit when tools can spend money, unlock hardware, or alter data. Every response includes a confidence value, and a threshold can send uncertain work elsewhere. That is a routing signal, not a safety boundary. The application still has to validate arguments, restrict permissions, log actions, and ask for approval where a mistaken call would hurt.
The 256-token window favors commands over conversation
The README specifies a 256-token sliding window and keeps tool information in pinned key-value slots. A retrieval head selects the top 5 tools for each turn when the catalog is large. This makes sense for commands such as setting a light or extracting invoice fields. It is a poor match for a long support chat whose answer depends on details mentioned many turns earlier. You can retain state outside the model, but then your own code decides what comes back into that small window.
Schemas do real work here. Needle compiles argument types, required fields, choices, and value constraints into its decoding grammar instead of asking the model to imitate JSON. That reduces malformed output, though it cannot prove that a valid city, amount, or device name matches the user's intent. Tool descriptions matter because the model reads them to select a function. Teams adopting Needle should treat those descriptions and schemas as application code, with review and regression cases, rather than loose prompt copy.
What happened when we ran it
Our sandbox installed commit 571fcd6 in 39 seconds. It pulled 66 packages and occupied 683 MB on disk, far more than the 14 MB inference binary discussed in the README because our measurement covers the Python environment. The build completed successfully in 9 seconds. Pip-audit reported 0 known vulnerabilities in the installed packages. The repository itself had 44 files, about 4,751 lines of source, and a 1.4 MB checkout.
Tests ended with exit code 1 after 55 seconds. Pytest reported 39 passed, 1 failed, and 5 skipped out of 40. test_pydantic_model_schema failed at its first line because Python could not import pydantic. The log does not say why that package was absent, so we cannot claim a packaging cause. It does show that the documented typed-extraction path was not fully covered by a passing suite in our fresh Python 3.12 Debian container.
The repository had 1 CI workflow and a tests directory, but no Dockerfile. That is reasonable for a package whose runtime engine is downloaded per platform, yet it leaves container packaging to the adopter. An air-gapped deployment needs the engine archive staged in advance, using the offline steps linked from the README. Data synthesis is a different path: it calls OpenRouter by default and therefore needs a key or a compatible gateway URL.
Quantized fine-tunes need an export-level evaluation
Fine-tuning uses LoRA against the frozen base checkpoint. The documented flow creates or supplies JSONL examples, trains an adapter, then merges and quantizes it into a .cact file. JAX can use NVIDIA or Apple Silicon acceleration through separate extras. The default export follows the checkpoint's bit map, with a 4-bit fallback, while --bits 2 requests a smaller result. The engine can load the tuned file without recompilation.
Open issue #91 gives a specific reason not to stop at validation loss. Its reporter says one adapter behaved correctly through the float JAX path, called almost nothing after a 2-bit export, and fired too often after a 4-bit export. Those figures belong to that reporter's evaluation, not ours, and the issue remains a report rather than a universal result. Still, the operational lesson is sound: run the same held-out commands through the merged float model and the exact file that will ship.
Python is ready first, while mobile integration is uneven
The README's clearest route is pip install cactus-needle, followed by a first engine download from Hugging Face. Once cached, inference needs no network. The playground starts a local web interface, loads the model before serving, and can hand back a fine-tuned archive. Developers can also download named platform runners and use the weights-agnostic engine. This covers experimentation and Python applications without much ceremony.
Native product integration needs more homework. Issue #92 asks for a Swift SDK, and issue #90 describes trouble using the distributed C header and static library because the caller could not interpret needle_init return codes or find the implementation source. Those reports do not prove every native path is broken. They do show that the Python examples are better documented than direct iOS or C++ embedding. If the product requirement begins with Swift, validate that interface before building the model workflow around it.
Fresh development does not settle the deployment risks
GitHub showed 9,304 stars, an Apache-2.0 license, and 27 open issues and pull requests. The last push was August 24, 2026, two days before this review, so the project is active. GitHub returned no latest release record through its releases API. That means adopters should pin a package version, engine file, and commit rather than infer a stable release cadence from repository activity alone.
Needle earns a trial when a narrow local model removes a cloud dependency or makes an embedded feature possible. The 39-second install and 9-second build lower the cost of testing it. The failed Pydantic test, short context window, native-interface questions, and quantized fine-tune report set the acceptance work: validate typed extraction, measure decisions on your tools, test the exported file, and run it on the actual device before granting it any authority.