Colibri added 2,035 GitHub stars in MrKeyoor's latest daily snapshot for making a 744-billion-parameter model fit around 16 GB of RAM. The missing line in that pitch is a 372 GB model on disk and a lot of waiting. The project's own performance table puts its 25 GB hardware floor at 0.05 to 0.1 token per second. At that rate, one generated token takes 10 to 20 seconds. Colibri matters because it moves an impossible memory problem onto storage, where developers can at least measure it.
That distinction helps explain the sudden attention. Colibri does not shrink every weight into system memory. Its runtime places weights across VRAM, RAM and NVMe, then moves the experts needed for each token closer to the processor. Developers get a way to study and run sparse models on hardware that would normally reject them at load time. They also inherit storage latency as part of every decoding decision.
Sparse activation creates the opening
Mixture-of-experts models contain many specialist feed-forward blocks, while a router selects only a subset for each token. In Colibri's reference GLM setup, the model has 744 billion parameters but activates about 40 billion per token. Only about 11 GB of routed expert weights change from one token to the next, according to the project's architecture explanation. That gap between total and active parameters gives the runtime something useful to schedule.
Colibri keeps the dense portion, including attention, embeddings and shared experts, resident in RAM at int4 precision. The project sizes that portion at about 17 billion parameters and 9.9 GB. Another 19,456 routed experts, roughly 19 MB each, occupy about 370 GB on disk. A per-layer cache holds recent experts, while recorded routing history can pin frequently used ones in faster memory. The README describes the result as placement: the selected model stays the same while the location of a weight changes.
That last promise needs careful wording. Placement is meant to affect speed without altering routing or precision inside a given Colibri container. The container itself may already be quantized. Colibri's quality notes report a mean normalized accuracy of 62.5% across small zero-shot HellaSwag, ARC and MMLU runs for the int4 reference container. An OLMoE comparison measured an 8.2 percentage-point loss from fp16 to int4 under the same harness. Disk streaming preserves the chosen container's computation; it does not erase quantization cost.
Storage becomes part of inference
A cache miss means reading expert matrices during generation, so Colibri works hard to avoid paying for the same bytes twice. The runtime documentation says it stores each expert's three matrices together, reads them with one pread, overlaps missing-expert loads with resident computation and prefetches one layer ahead. The project measured the next layer's routing as 71.6% predictable in its test. That number comes from Colibri's own experiments, and workloads with different routing patterns may behave differently.
The cache also learns from use. A .coli_usage file records which experts a workload selects, allowing later runs to pin the hottest weights. Colibri's open-hypotheses table says those learned pins improve repeated workloads but can overfit a prompt. The maintainers are asking for held-out, cross-session comparisons across coding, chat and long-context jobs. That is a more useful caveat than a single warmed-up demo because cache history is part of the product's performance.
Hardware changes the result by orders of magnitude. Colibri's published ladder reports about 1.8 tokens per second on a warm 128 GB CPU-only desktop. A six-RTX-5090 system with the experts fully resident reaches 5.8 to 6.8 tokens per second. The 25 GB floor falls to 0.05 to 0.1. Those are project and contributor measurements on different machines, rather than a controlled comparison, but they make the boundary visible: opening the model is possible on a small host; interactive speed still depends on fast residency and memory bandwidth.
A second SSD can help when it sits on an independent controller. Colibri can keep a byte-identical model copy on another drive and distribute reads according to measured bandwidth. One Threadripper test rose from 0.80 to 1.10 tokens per second, a 37.5% gain. A different file-level split gained only 5.5%, because spreading whole shards did not parallelize reads inside an expert. The storage layout matters as much as the second drive's label.
A small runtime still carries a large operational load
The inference engines are written in C and avoid BLAS or Python at runtime. The surrounding coli launcher and API gateway do use Python, and source builds need GCC or Clang with OpenMP. Version 1.11.0 supports nine model families through separate model-specific engines, including GLM, Inkling, Kimi, DeepSeek, Qwen and OLMoE variants. This is focused compatibility, not an assurance that an arbitrary MoE checkpoint will load.
The first sensible commands inspect the machine before attempting a long model setup. Colibri's README exposes a placement plan and a deeper read-only check:
COLI_MODEL=/path/to/model ./coli plan
COLI_MODEL=/path/to/model ./coli doctor --deep
Those commands cannot supply missing capacity. The documented weights range from about 7 GB for OLMoE to roughly 1.6 TB for Kimi K3, while the GLM reference uses about 372 GB. Our review of Colibri covers the setup reality: a fresh sandbox installed 34 Python packages and completed its measured build in 10 seconds, but it did not load model weights or generate a token. The tiny engine and the usable deployment are very different sizes.
Read the benchmark conditions
Colibri's benchmark file is unusually specific about hardware, cache state, commands and failed experiments. It also mixes maintainer runs with community reports across many model versions and hosts. The contribution protocol asks testers to record the commit, model container, exact command, prompt, cache condition, throughput and expert hit rate. Readers should compare rows only when those conditions match. A warm figure from one model cannot stand in for a cold run on another.
The latest release gives a useful example of that measurement culture. The v1.11.0 notes say two attempts to hide DeepSeek V4.1 expert reads behind matrix multiplication measured worse and were removed. The same release added the ninth model family after 56 pull requests and fixed five reported bugs across four engines. Fast iteration is producing real work, though it also means operators need to pin versions and retest their own path.
The native parser remains an attack surface even though the engine has no runtime dependencies. Colibri 1.6.2 patched six privately reported memory-safety flaws reachable through malicious model data or the Kimi service input path. The release added checks at the trust boundary and recommended upgrading for anyone loading models from untrusted mirrors or exposing that protocol to untrusted clients. Model files are security-sensitive inputs when native code parses shapes, offsets and tensor layouts from them.
Colibri therefore makes the most sense as an inference-systems project for developers who can measure storage, verify a specific model container and accept hardware-dependent latency. Its Apache-2.0 repository supplies chat, server and web entry points, but the project explicitly offers no speed service level. Teams needing predictable multi-user serving still have to test queueing, cancellation, malformed inputs and sustained load on the exact engine they intend to expose.
What to watch
Colibri's next useful result should put its learned cache through a held-out workload. The project already lists cross-session placement, cold-cache dual-drive runs and automatic hardware planning as open experiments. If independent runs show that routing history transfers across real workloads without sacrificing correctness, SSD-backed experts could become a repeatable way to inspect models that otherwise require a rented server. If the gains disappear outside warm prompts, the 2,035-star day will have bought developers access more than speed.