FlashStates cuts repeated prompt processing by saving a hybrid model's recurrent checkpoints together with the attention cache they need. In Sandipan Kundu's independent H200 experiment, whole-run output throughput improved by 6.7% and 26.9% across two synthetic workloads, but both failed the exact-output-parity threshold. Kundu's results make this worth investigating for inference engineers, not an automatic production replacement.
Use it if you operate hybrid models and can evaluate cache behavior and output correctness on your own traffic. Skip it if you need a supported, predictable serving stack without maintaining research changes: the README explicitly calls FlashStates research code. Our install and build checks succeeded for the detected Mooncake subproject; our later deployment attempt stopped at a safety check.
What people use FlashStates for
The authors replay coding-agent traffic, Kundu tests controlled synthetic sessions, and Auryn adapts the design to local inference. Those are different kinds of experience, and you should choose the one closest to your workload.
| Who used it | What they ran | What they reported |
|---|---|---|
| FlashStates authors | Public coding-agent sessions across three hybrid models, two B200 GPUs per engine | Up to 3.02× decode throughput, 3.14× prefill throughput and 88% lower p90 time to first token versus vLLM, as workload-specific maxima |
| FlashStates authors | Kimi-K3.1-Pro, eight GB300 GPUs, DSpark speculative decoding, decode context parallelism and a 640 GiB Mooncake store | Peak output throughput of 8,683 versus 6,377 tokens per minute per GPU, with p90 first-token waits of 2.0 versus 9.1 seconds, in the reported measurement window |
| Sandipan Kundu | Qwen3.8-27B-FP8 on one H200, four sessions fitting in GPU cache | Median later-turn recomputation fell from 1,500 to 301 tokens and whole-run output throughput rose 6.7% in his mechanism experiment |
| Sandipan Kundu | 123 synthetic sessions under GPU-cache pressure, without Mooncake | Whole-run output throughput rose 26.9%, but the median-recomputation target failed in his pressure experiment |
| Auryn | A FlashStates-inspired exllamav3 adaptation, DeepSeek-V4.1-Flash on four RTX 3090s | Follow-up prefill fell from 4.06 to 0.08 seconds and wall time from 20.58 to 0.44 seconds in Auryn's example |
Auryn's X article is useful because it follows the idea into a different engine and smaller cards. It is a separate implementation, with CPU and disk cache tiers, rather than a reproduction of the vLLM fork. Its reported gains belong to that implementation and example.
What happened when we ran it
Our November 20 sandbox install succeeded in 36 seconds and its build succeeded in 2 seconds at commit 6d3503e; the detected Python project was under ./third_party/mooncake/. Those are repository setup results, not inference measurements.
The following day's hands-on attempt stopped before a demonstrated serving workload. The safety step returned: "unsafe repo: none of compose.yaml, compose.yml, docker-compose.yaml, docker-compose.yml exists (rule: a compose file is required)". The runner required Compose, so it rejected this checkout. That finding describes the runner's requirement, not a failure of FlashStates' documented Docker command.
| Our recorded check | Outcome | What you can conclude |
|---|---|---|
| November 20 dependency install | Succeeded in 36 seconds; 51 dependencies recorded; 124 MB on disk | The detected subproject's install step completed |
| November 20 build | Succeeded in 2 seconds | The recorded build step completed |
| November 20 tests | No tests script or target; skipped | There is no passing test-suite result |
| November 21 deployment safety check | FAILED: no accepted Compose filename found | The hands-on attempt stopped at the safety gate |
| November 21 cleanup | teardown: ok | The runner recorded successful teardown |
The November 21 run summary says deployment used the project's own Docker Compose, with ports bound to 127.0.0.1. That conflicts with the failed safety step. We count this as a stopped attempt, not a working deployment or a completed model task.
The November 20 sandbox configuration was 3 CPUs and 8 GB RAM, using image label ghcr.io/astral-sh/uv:python3.12-bookworm. The label alone doesn't identify the runtime or operating-system versions actually used. The scan counted 11,459 files, approximately 3,099,588 source lines and a 245.3 MB checkout, plus one CI workflow file, no Dockerfile and no tests directory. These are our scan observations. They don't overturn the README's separate description of a B200 container image and its build files.
Where people get stuck: placement alone can waste cache
Saving a checkpoint at the right token is only useful if its matching attention cache survives. The project describes retaining and evicting those dependencies together, priced by the prompt-processing time saved per block of memory freed.
Prefill means processing an incoming prompt before generating a reply. KV means the keys and values attention layers save for earlier tokens. A recurrent layer instead updates a running internal state; to resume an older prefix, the engine needs the state saved at that prefix's endpoint. A later state cannot simply be shortened like an attention cache.
The Qwen ablation in the alphaXiv overview shows why checkpoint placement and cache policy belong together. At 32 sessions, exact checkpointing alone produced 26.0K output tokens per minute per GPU, below vLLM's 30.1K. The complete system recovered to 30.4K, but that small advantage was not statistically significant in the single-run ablation. Retaining more checkpoints can displace something more useful.
| Obstacle | Documented response | Status or boundary |
|---|---|---|
| A B200-sized cache configuration doesn't fit a smaller card | Adjust --kv-cache-memory-bytes, keeping the same budget for each engine |
An explicit installation-guide instruction |
| The optional GLM prefill-export kernel targets B200/SM100 | Other GPUs use FlashKDA's existing checkpoint path; Qwen and Nemotron use different export backends | The GPU and model paths differ |
| Auryn's partial-page checkpoints lack CPU cache capacity | Enable CPU cache; otherwise a warning accompanies fallback to page-aligned checkpoints | A requirement of the exllamav3 adaptation |
| Speculative drafts advance recurrent state beyond accepted tokens in Auryn's adaptation | Settle state through reject_remainder before checkpointing |
Auryn reports a bit-identical 64-token continuation after the fix |
These are different remedies for different paths. Copying a kernel instruction or cache setting from another model can miss the actual prerequisite.
Under cache pressure, averages hide a long wait
Kundu's pressure experiment improved whole-run throughput by 26.9%, yet its median later-turn first-token wait was still 98.785 seconds. His client-side TTFT measurement includes queueing and HTTP overhead, so that result describes a heavily loaded service rather than just kernel speed.
His four-session experiment is easier to interpret: each turn added 300 synthetic tool-result tokens, and FlashStates recomputed a median of 301 tokens. That is close to processing just the new input. Under 123-session pressure, the median uncached count barely changed, from 32,400 to 31,900 tokens. Better reuse for a smaller group pulled total recomputation down by 28.1% without rescuing the typical request.
Kundu defines whole-run output throughput as generated tokens divided by the full benchmark time, including prefill and waiting. Don't equate its 6.7% or 26.9% improvement with a change in per-request decode speed. Likewise, a token-weighted cache-hit rate describes the fraction of prompt tokens reused, not the fraction of requests served entirely from cache.
For your own comparison, keep cache capacity, model files, scheduler settings and workload shape equal. Record the distribution of first-token waits alongside aggregate output. A service can complete more total work while an individual agent still waits too long to be useful.
FlashStates vs vLLM, SGLang and local exllamav3
Stock vLLM is the directly measured baseline in Kundu's H200 comparison; SGLang and exllamav3 answer different adoption questions. There is no single result here that ranks every serving engine.
| Alternative | Why you might choose it | What changes the decision |
|---|---|---|
| Stock vLLM 0.33.0 | You want the baseline without this research overlay | Kundu reports 6.7% to 26.9% better whole-run throughput with FlashStates, alongside failed parity checks |
| SGLang | You already operate SGLang and want to follow its native hybrid-cache work | Issue #4 reports merged state-cost eviction and checkpoint-level controls, with unit eviction under review |
| Marconi | You are comparing selective hybrid-state retention designs | The alphaXiv overview distinguishes Marconi's tunable priority coefficient from FlashStates' policy, which still fits prefill costs per model |
| HiState | You are investigating memory tiers and adaptive checkpoint intervals | The overview reports no head-to-head comparison, so it supplies no winner |
| exllamav3 with Auryn's patches | You run local inference and can evaluate CPU/disk cache behavior | The adaptation has no Mooncake connector or dedicated mid-prefill export kernel |
My preference for an existing deployment is to investigate the mechanism in the engine you already own before taking on a fork. The independent comparison gives vLLM operators a concrete reason to evaluate FlashStates. SGLang operators should track the individual merged features, rather than treating the whole design as either present or absent.
Source setup needs a GPU plan, not just a Python environment
The source guide calls for Linux x86_64, CUDA 13 drivers, Python 3.12 and uv. It separates the vLLM overlay, optional Mooncake store and replay client, so a small successful subproject install doesn't describe the full serving setup.
The README's ready-made images target B200 and GB300 configurations. Its serving example mounts model files read-only, enables prefix caching and supplies explicit checkpoint, eviction and model-cost settings. Clients retain the vLLM OpenAI-compatible API. You still need the model files and a configuration appropriate to the hardware.
Mooncake is an optional host-memory tier, not a prerequisite for Kundu's GPU-only results. Choosing it introduces a separate store configuration. The source guide also keeps the official baseline in a separate environment, which matters when your question is whether FlashStates improves on stock vLLM rather than on another mixture of patches.
Auryn's local implementation adds another practical lesson: dynamic content early in a prompt breaks exact-prefix reuse. A changing date or session identifier before a large static block can make that block expensive again. Its persisted weights also differ from persisted model tensors: warming after restart replays stored token chains and pays prefill work again.
Security defaults: the example uses host networking
The README's Docker command uses --network host and --ipc=host, alongside a read-only model mount. Those example settings deserve an infrastructure review before you expose the server to clients.
Treat the command as a model-serving example, then decide where your gateway, access controls and network boundaries belong. A read-only model mount controls writes to that directory; it doesn't answer who can send requests to the service. The failed Compose attempt's localhost-binding summary is no evidence of the published Docker command's default exposure.
What the other reviews get wrong
AINEWS Radar's account misses Kundu's independent H200 comparison, while Satyajit's older analysis predates an upstream vLLM decode-checkpoint change. Both distinctions affect what you should conclude today.
AINEWS Radar says its account contains no independent benchmarks or reproduction. Kundu's experiment supplies a controlled independent comparison, but it doesn't reproduce the authors' B200 headline: the GPU, model and workload differ.
Satyajit's November 9 analysis describes vLLM as never saving recurrent checkpoints during decode. The November 19 vLLM RFC says main already retains the latest block boundary crossed during decode. Retention at finer prefix-match boundaries is the proposal, not the entire idea of retaining decode state.
Two project statements also need narrower readings. The landing page says outputs stay bit-identical. Kundu's failed exact-output comparison limits that as a deployment-wide promise, without isolating checkpointing as the cause or disproving the narrower kernel claim. The install guide's options-off equivalence has exceptions: the fork commit documents three always-on fixes, involving unwritten Mamba state columns, the GLM MTP layer and overlaid KV copies.
Kirill Pokidov's LinkedIn summary goes too far in saying redundant prefill is eliminated. Kundu still measured 20.09 million later-turn uncached tokens under pressure. FlashStates reduces repeated work; it doesn't make cache capacity unlimited.
Finally, Satyajit's analysis separates serving measurements from the pricing overlay behind profit and user-cost claims. Faster token serving isn't by itself a measured reduction in your bill or an increase in your margin.
Questions people ask
SGLang support is in progress: issue #4 lists merged components and unit eviction under review, rather than a completed supported FlashStates edition for SGLang. The maintainer's reply is the useful starting point.
Is there a supported SGLang version of FlashStates that I can test?
The maintainer reports --radix-eviction-policy state_cost and --hybrid-cache-checkpoints-level merged into main. Unit eviction, --enable-mamba-unit-eviction, is under review and off by default in that proposal. Follow the roadmap linked from issue #4 for the individual pieces rather than assuming a full supported port.
How much of the reported 3× gain comes from dependency-aware retention versus exact-turn checkpoints?
There is no defensible universal split. The Qwen ablation rises from 24.1K output tokens per minute per GPU for vLLM to 34.2K with checkpointing, 36.8K with admission and 38.8K with the complete system at 48 sessions. Those model-specific staged results don't allocate the separate GLM headline. Kundu enabled four features together, so his experiment doesn't isolate their individual contributions.
Does FlashStates retain its advantage with shorter sessions?
The shorter-session question is open in Colin-Mario Köse's comment on Sudip Roy's post. Kundu's four-session result demonstrates a benefit when sessions fit, but changing concurrency isn't a controlled session-length experiment. His fixed eight-turn design gives no general short-session answer.
What happens to throughput between 40 and 48 concurrent agents?
The discussion asks for the curve between those loads; it doesn't measure the intervening behavior. Kundu compares four and 123 sessions and explicitly says a session-count sweep is needed. Don't interpolate an operational limit from those two settings.
At what load does throughput start falling, and how much cache headroom is needed?
There is no portable agent-count or cache-headroom rule. The authors' profiling shows capacity misses growing from about 40 sessions in their configuration, while Kundu says his overloaded setting cannot locate peak capacity. The project's capacity discussion belongs to its coding traces and equal-memory comparison, not every GPU.
Should upstream vLLM expose a separate decode checkpoint-retention interval, and what should it be called?
The November 19 RFC proposes a separate decode interval, with prefix_cache_decode_retention_interval as an example name still to be agreed. It distinguishes default retention at the last prefix-match boundary, periodic states with a positive interval and compatibility behavior with None. That is a design proposal seeking feedback, not a settled API recommendation.
Should the exllamav3 adaptation derive GDS coefficients per model rather than require manual configuration?
Auryn explicitly asks reviewers that question; his article doesn't resolve it. GDS, GreedyDual-Size, ranks retained checkpoints by saved processing cost relative to storage. The adaptation exposes manually configured coefficients, so automatic derivation is a review question rather than a shipped feature.
What we have not tested yet
Our successful subproject setup checks and stopped deployment attempt leave GPU inference, model loading, answer quality and production reliability untested by us. We did not reproduce the B200 headline, Kundu's H200 experiment or Auryn's local adaptation.
Kundu's random-token workload evaluates serving behavior, not coding ability or task completion. Its exact-output match rates were 71.43% and 80.92%, both below the predefined 98% bar. Some first-turn outputs already differed, before checkpoint reuse could explain them. The author lists numerical sensitivity, batching, scheduling and resume behavior as possible causes, and reports warnings about uncalibrated FP8 KV scaling factors. These are unresolved causes, not a diagnosis or a measured quality loss. Once outputs diverged, subsequent prompt contents differed too, although sequence lengths and workload shape stayed matched.
We have no security audit, load sweep on the intended deployment, restart-recovery exercise or evidence of broad customer adoption. The upstream RFC also leaves pipeline parallelism and some offload connectors outside its initial coverage. A useful next evaluation pairs a load sweep with output checks on representative tasks, keeping model revision, cache budget and engine version fixed. Decide from the first-token waits and completed correct tasks your users actually receive.


