mrkeyoor.com_
Tue 11 Aug 13:32 UTC
LLM Toolsevaluationupdated 11 Aug 2026

needle

Needle is a very small language model built to choose tools, fill their arguments, and extract structured records on devices with limited memory. The project packages its 45-million-parameter Needle 2 model in a 14 MB custom engine, then exposes it through Python with schema-constrained JSON output and confidence scores.

Verdict

Needle is one of the most interesting options for a narrow class of offline agents where memory use matters more than broad language ability. Its constrained schemas, retrieval over large tool catalogs, and explicit low-confidence path are better product primitives than asking a tiny model to imitate a chatbot. Prototype it, but do not commit a shipping workflow until your own tool set passes a serious evaluation and the current quickstart regression is resolved.

Setup3/5One pip command, but a current report breaks the headline example
Docs4/5Detailed APIs and tuning guide, with mobile and evaluation gaps
Community3/5Fast early interest and active changes, but a small young issue base
Maturity2/5Promising specialist model with no GitHub release and fresh regressions

Who it’s for

Python developers building private, offline tool calling for appliances, robots, wearables, or low-memory computers.
Product teams whose commands map cleanly to typed functions and JSON schemas.
Applications that can route low-confidence requests to a person or a larger model.
Researchers experimenting with tiny models, constrained decoding, tool retrieval, LoRA tuning, and custom quantization.

Who it’s NOT for

Teams looking for a general chat assistant: the README says Needle solves tasks as function calls, returns an empty call for unsupported requests, and has no free-text fallback for off-topic input.
Applications that need a long exact conversation history: the model uses a 256-token sliding window, even though its tools remain pinned in memory.
Developers who need the advertised quickstart to be dependable today: issue 46 reports that the README's weather example returns no tool call on both macOS and Linux.
Mobile teams expecting a ready React Native, Android, or iOS integration in this repository: the documented product surface is a Python package and custom engine, not a mobile application SDK.
Anyone who needs standard model artifacts throughout fine-tuning: the documented training path writes and reads pickle-based checkpoint files before exporting the final custom binary, and issue 36 raises the security concern explicitly.

Setup reality

The first inference trial is supposed to be one pip install cactus-needle followed by a short Python script. The native engine is downloaded from Hugging Face on first use and cached, so inference later runs offline, but the package also declares JAX, Flax, Optax, NumPy, SentencePiece, and Hugging Face Hub dependencies. A current cross-platform report says the headline weather example does not actually call its tool, so budget time for behavioral testing rather than judging success by import alone. Fine-tuning is substantially heavier: prepare JSONL examples, optionally pay for OpenRouter-generated data, run LoRA training, merge the adapter, quantize it, and export a .cact file.

A specialist model with a sensible job

Needle 2 is not trying to be a tiny replacement for a conversational model. It has one main job: read a request, select a declared function, and produce arguments that fit its schema. Structured extraction is treated as the same task with one available function. Many local assistants do not need essays or broad world knowledge. They need to turn "dim the living room to 30" into a safe, predictable command.

Cactus Compute says the 45-million-parameter model is compressed into a single 14 MB binary and uses about 28 MB of memory for a full session. The repository contains the Python interface, inference path, LoRA fine-tuning workflow, and exporter. A native engine and weights are fetched from Hugging Face on first use, cached, and then run without a network connection. For an appliance, wearable, smart-home controller, or robot, that size and offline behavior could matter more than fluent conversation.

The MIT license is friendly, although the Python package metadata currently says Apache 2.0 while the repository's actual license file says MIT. That inconsistency should be fixed.

Constrained output is the strongest idea

Needle compiles a byte-level grammar from each tool's JSON schema. The generated call therefore has to match declared types, enumerations, numeric ranges, string patterns, and array limits. A malformed JSON object or out-of-range value is blocked during decoding instead of being noticed after generation. Python decorators derive schemas from signatures and docstrings, while Field adds tighter constraints and Pydantic models provide typed extraction.

This is a meaningful improvement over parsing whatever a small model happens to write. It still does not prove that the selected tool or value is semantically correct. If the user asks for one recipient and the model chooses another valid recipient, grammar cannot save you. Applications must validate permissions, confirm high-impact actions, and test ambiguous language.

The confidence mechanism is designed for that uncertainty. Each answer includes a score based on a learned head and call-token probabilities. The README recommends choosing a threshold, acting above it, and escalating below it. Teams still need to calibrate that threshold on their own commands.

Large tool catalogs fit, long context does not

When five or fewer tools are declared, Needle renders all of them. With a larger catalog, a built-in retrieval head embeds the query and exposes only the five highest-scoring tools to the model. The decode grammar is rebuilt around that subset, so an unselected tool cannot be called. Embeddings can be persisted to avoid rebuilding an unchanged index. This gives a small model a route into a much larger action catalog without placing every schema in its prompt.

The cost is another failure boundary. A correct tool omitted by retrieval is unreachable, no matter how well the calling model understands the request. Test retrieval recall separately from argument accuracy. Similar tool names and overlapping descriptions deserve extra attention. The README says writing good tool descriptions is central to results, which puts real design work on the integrator.

Conversation memory is deliberately bounded. A 256-token sliding window limits memory growth, while tool definitions remain pinned as key-value sinks. That suits short command chains and device control, but it is not a substitute for a long support conversation or a complex planning agent. Important state should live in the application and be supplied as explicit facts or tool results.

Easy installation does not mean finished integration

The promised trial is short: install cactus-needle, decorate a Python function, create an agent, and run a request. The engine download happens once. Needle supports direct schemas, manual tool loops, a browser playground, and a helper for typed extraction. Python 3.9 or newer is declared, and platform-specific native libraries are selected for macOS, Windows, and Linux.

The package is heavier than the 14 MB model headline suggests. Its Python dependencies include JAX, JAXlib, Flax, Optax, NumPy, SentencePiece, and Hugging Face Hub. More importantly, issue 46, opened August 10, shows the exact weather quickstart returning a normal response with no function call on macOS. A second user reproduced it on Linux. That is a direct reason to test before designing around the API.

The mobile claim also needs careful reading. The model and Cactus engine target tiny devices, but this repository documents a Python package rather than end-to-end Android, iOS, or React Native integration. Mobile developers should evaluate the broader Cactus runtime and confirm application bindings.

Fine-tuning is accessible, but not trivial

Needle accepts JSONL examples pairing a query and tool schemas with expected calls. It can optionally synthesize data through OpenRouter, run LoRA fine-tuning, merge the adapter, quantize the result, and emit a custom .cact binary. A playground button wraps much of that workflow.

Good data remains the hard part. Synthetic examples need review, off-topic cases need explicit empty answers, and every sensitive action needs adversarial prompts. The intermediate workflow uses pickle checkpoint files. Issue 36 correctly warns that loading untrusted pickle data can execute code. The final runtime artifact is a different format, but training teams should still control checkpoint provenance and storage.

Young, active, and not yet settled

The repository was created in February 2026 and pushed on August 10. Recent merged work fixed JAX compatibility and false TPU detection, while open pull requests cover decoding speed and correctness. The GitHub count showed 28 open issues and pull requests, and the project had no GitHub release even though its package identifies itself as version 2.0.0.

Needle is ready for a focused evaluation, not blind adoption. Build a test set from real utterances, measure tool retrieval and arguments separately, set confidence thresholds from those results, and require confirmation for irreversible actions. If it clears that bar, very few alternatives offer this much local tool-calling machinery in such a small claimed footprint.

Alternatives

ProjectWhat it isPick it when
CactusThe broader Cactus runtime for quantized model inference on phones and other small devices.pick this instead when you need a mobile inference platform or models beyond Needle's narrow tool-calling contract.
llama.cppA widely used C and C++ runtime for running many quantized language models locally.pick this instead when ecosystem breadth, model choice, and established deployment tooling matter more than a 14 MB specialist model.
MLX Swift ExamplesApple's examples for running and integrating MLX models in Swift applications.pick this instead when your target is an Apple-native Swift app and you want direct MLX integration rather than a Python wrapper.

What people are saying

  1. [github-trending] cactus-compute/needle

Sources

  1. Needle repository and README
  2. README quickstart failure on macOS and Linux
  3. Needle pickle checkpoint security discussion
  4. Simple Attention Network paper