mrkeyoor.comAsk

FutureMLS-Lab/FlashStatestested 20 Nov 2026commit 6d3503e

FlashStates review

Our sandbox install took 36 seconds and the build took 2 seconds for the detected Mooncake subproject, while our later deployment attempt stopped at a Compose safety check.

  • Safety failed
  • Build 2 s
36 sto install
  • 51 packages
LLM Toolsevaluationupdated 21 Nov 2026
trackingstars / 7d

FlashStates changes how hybrid language models save and reuse conversation state between agent turns. It retains recurrent checkpoints together with their matching attention cache, reducing repeated prompt processing in its vLLM fork, with an optional Mooncake host-memory store.

Verdict

Our sandbox install took 36 seconds and the build took 2 seconds for the detected Mooncake subproject, while our later deployment attempt stopped at a Compose safety check. Use FlashStates if you operate hybrid models and can measure cache reuse and output correctness against stock vLLM. Skip it if you need a supported production replacement: the project calls itself research code, and Kundu's independent H200 comparison failed exact-output parity despite higher throughput.

FlashStates infographic: save and reuse conversation state between agent turns, plus our test result and who it suits
Tested 20 November 2026: the source install took 36 s and the build 2 s.

We ran it

Lab card: what happened when we ran FlashStates
Lab card from our sandbox run
Screenshot of FlashStates (github.com/FutureMLS-Lab/FlashStates)
github.com/FutureMLS-Lab/FlashStates
Install✓ · 36s51 packages · 124 MB
Build✓ · 2s
Testsn/ano test script
Repo11459 files~3,099,588 lines of source · 245.3 MB · 1 CI workflows

Answers from our run

Does FlashStates build from source?

Dependencies installed in 36 seconds (51 packages), and the build succeeded in 2 seconds. We cloned commit 6d3503e into a clean Debian container with 3 CPUs and no project-specific setup.

Does FlashStates have tests you can run?

Not through a standard command: the project exposes no test script or target that our harness could run.

Who should not use FlashStates?

Operators who need a supported production replacement: the README labels FlashStates research code and says its options may change.

What are the alternatives to FlashStates?

vLLM, SGLang, exllamav3. Our sandbox install took 36 seconds and the build took 2 seconds for the detected Mooncake subproject, while our later deployment attempt stopped at a Compose safety check.

Setup4/5Our source checkout installed in 36 s and built in 2 s with 51 dependencies, but we did not run a README task.Editor’s note: The recorded subproject install and build succeeded, but full GPU serving setup was not reproduced and the later deployment runner stopped at its Compose requirement.
Docs4/5The README has an install section (“Getting Started”) and usage in the same section; we did not run its quickstart ourselves.Editor’s note: The README and source guide explain hardware paths and flags; the options-off equivalence needs the always-on-fix exceptions documented by the fork commit.
Community3/5Last push on 2026-11-21 and 282 stars.Editor’s note: Issue #4 has a maintainer roadmap reply and independent experiments exist, but this sample cannot measure general support quality or customer adoption.
Maturity1/5Latest release v0.1.0 on 2026-11-09 (pre-1.0) and repository created 2026-11-09.Editor’s note: The README labels this research code. Our checks do not assess production reliability, and the independent output-parity failure needs investigation.

How we score

Who it’s for

Inference engineers serving hybrid models with growing agent conversations who can compare cache reuse and outputs on their own workloads.
vLLM operators willing to evaluate the research fork against a separately configured stock baseline.
Researchers studying checkpoint placement, admission and eviction under memory pressure.

Who it’s NOT for

Operators who need a supported production replacement: the README labels FlashStates research code and says its options may change.
Teams requiring demonstrated exact-output equivalence: Kundu's independent comparison failed its parity threshold in both workloads.
SGLang users expecting a complete supported port: issue #4 describes support in progress, with unit eviction still under review.

Setup reality

Our November 20 run at commit 6d3503e recorded a successful install in 36 seconds, 51 dependencies and 124 MB on disk, followed by a successful 2-second build. The detected Python project was ./third_party/mooncake/. Tests were skipped because no script or target was found; these checks don't demonstrate a running inference server.

The source guide requires Linux x86_64, CUDA 13 drivers, Python 3.12 and uv, plus model files and hardware-specific cache settings. The vLLM overlay is separate from the optional Mooncake store and replay client. Our November 21 deployment attempt stopped because the runner required a Compose file and found none; its summary's deployment claim conflicts with that failed step.

Our sandbox configuration had 3 CPUs, 8 GB RAM and image label ghcr.io/astral-sh/uv:python3.12-bookworm. Actual runtime and operating-system versions require the raw toolchain logs. B200 reference cache budgets need adjustment for smaller GPUs, and the optional GLM export kernel targets B200/SM100.

FlashStates cuts repeated prompt processing by saving a hybrid model's recurrent checkpoints together with the attention cache they need. In Sandipan Kundu's independent H200 experiment, whole-run output throughput improved by 6.7% and 26.9% across two synthetic workloads, but both failed the exact-output-parity threshold. Kundu's results make this worth investigating for inference engineers, not an automatic production replacement.

Use it if you operate hybrid models and can evaluate cache behavior and output correctness on your own traffic. Skip it if you need a supported, predictable serving stack without maintaining research changes: the README explicitly calls FlashStates research code. Our install and build checks succeeded for the detected Mooncake subproject; our later deployment attempt stopped at a safety check.

What people use FlashStates for

The authors replay coding-agent traffic, Kundu tests controlled synthetic sessions, and Auryn adapts the design to local inference. Those are different kinds of experience, and you should choose the one closest to your workload.

Who used it What they ran What they reported
FlashStates authors Public coding-agent sessions across three hybrid models, two B200 GPUs per engine Up to 3.02× decode throughput, 3.14× prefill throughput and 88% lower p90 time to first token versus vLLM, as workload-specific maxima
FlashStates authors Kimi-K3.1-Pro, eight GB300 GPUs, DSpark speculative decoding, decode context parallelism and a 640 GiB Mooncake store Peak output throughput of 8,683 versus 6,377 tokens per minute per GPU, with p90 first-token waits of 2.0 versus 9.1 seconds, in the reported measurement window
Sandipan Kundu Qwen3.8-27B-FP8 on one H200, four sessions fitting in GPU cache Median later-turn recomputation fell from 1,500 to 301 tokens and whole-run output throughput rose 6.7% in his mechanism experiment
Sandipan Kundu 123 synthetic sessions under GPU-cache pressure, without Mooncake Whole-run output throughput rose 26.9%, but the median-recomputation target failed in his pressure experiment
Auryn A FlashStates-inspired exllamav3 adaptation, DeepSeek-V4.1-Flash on four RTX 3090s Follow-up prefill fell from 4.06 to 0.08 seconds and wall time from 20.58 to 0.44 seconds in Auryn's example

Auryn's X article is useful because it follows the idea into a different engine and smaller cards. It is a separate implementation, with CPU and disk cache tiers, rather than a reproduction of the vLLM fork. Its reported gains belong to that implementation and example.

What happened when we ran it

Our November 20 sandbox install succeeded in 36 seconds and its build succeeded in 2 seconds at commit 6d3503e; the detected Python project was under ./third_party/mooncake/. Those are repository setup results, not inference measurements.

The following day's hands-on attempt stopped before a demonstrated serving workload. The safety step returned: "unsafe repo: none of compose.yaml, compose.yml, docker-compose.yaml, docker-compose.yml exists (rule: a compose file is required)". The runner required Compose, so it rejected this checkout. That finding describes the runner's requirement, not a failure of FlashStates' documented Docker command.

Our recorded check Outcome What you can conclude
November 20 dependency install Succeeded in 36 seconds; 51 dependencies recorded; 124 MB on disk The detected subproject's install step completed
November 20 build Succeeded in 2 seconds The recorded build step completed
November 20 tests No tests script or target; skipped There is no passing test-suite result
November 21 deployment safety check FAILED: no accepted Compose filename found The hands-on attempt stopped at the safety gate
November 21 cleanup teardown: ok The runner recorded successful teardown

The November 21 run summary says deployment used the project's own Docker Compose, with ports bound to 127.0.0.1. That conflicts with the failed safety step. We count this as a stopped attempt, not a working deployment or a completed model task.

The November 20 sandbox configuration was 3 CPUs and 8 GB RAM, using image label ghcr.io/astral-sh/uv:python3.12-bookworm. The label alone doesn't identify the runtime or operating-system versions actually used. The scan counted 11,459 files, approximately 3,099,588 source lines and a 245.3 MB checkout, plus one CI workflow file, no Dockerfile and no tests directory. These are our scan observations. They don't overturn the README's separate description of a B200 container image and its build files.

Where people get stuck: placement alone can waste cache

Saving a checkpoint at the right token is only useful if its matching attention cache survives. The project describes retaining and evicting those dependencies together, priced by the prompt-processing time saved per block of memory freed.

Prefill means processing an incoming prompt before generating a reply. KV means the keys and values attention layers save for earlier tokens. A recurrent layer instead updates a running internal state; to resume an older prefix, the engine needs the state saved at that prefix's endpoint. A later state cannot simply be shortened like an attention cache.

The Qwen ablation in the alphaXiv overview shows why checkpoint placement and cache policy belong together. At 32 sessions, exact checkpointing alone produced 26.0K output tokens per minute per GPU, below vLLM's 30.1K. The complete system recovered to 30.4K, but that small advantage was not statistically significant in the single-run ablation. Retaining more checkpoints can displace something more useful.

Obstacle Documented response Status or boundary
A B200-sized cache configuration doesn't fit a smaller card Adjust --kv-cache-memory-bytes, keeping the same budget for each engine An explicit installation-guide instruction
The optional GLM prefill-export kernel targets B200/SM100 Other GPUs use FlashKDA's existing checkpoint path; Qwen and Nemotron use different export backends The GPU and model paths differ
Auryn's partial-page checkpoints lack CPU cache capacity Enable CPU cache; otherwise a warning accompanies fallback to page-aligned checkpoints A requirement of the exllamav3 adaptation
Speculative drafts advance recurrent state beyond accepted tokens in Auryn's adaptation Settle state through reject_remainder before checkpointing Auryn reports a bit-identical 64-token continuation after the fix

These are different remedies for different paths. Copying a kernel instruction or cache setting from another model can miss the actual prerequisite.

Under cache pressure, averages hide a long wait

Kundu's pressure experiment improved whole-run throughput by 26.9%, yet its median later-turn first-token wait was still 98.785 seconds. His client-side TTFT measurement includes queueing and HTTP overhead, so that result describes a heavily loaded service rather than just kernel speed.

His four-session experiment is easier to interpret: each turn added 300 synthetic tool-result tokens, and FlashStates recomputed a median of 301 tokens. That is close to processing just the new input. Under 123-session pressure, the median uncached count barely changed, from 32,400 to 31,900 tokens. Better reuse for a smaller group pulled total recomputation down by 28.1% without rescuing the typical request.

Kundu defines whole-run output throughput as generated tokens divided by the full benchmark time, including prefill and waiting. Don't equate its 6.7% or 26.9% improvement with a change in per-request decode speed. Likewise, a token-weighted cache-hit rate describes the fraction of prompt tokens reused, not the fraction of requests served entirely from cache.

For your own comparison, keep cache capacity, model files, scheduler settings and workload shape equal. Record the distribution of first-token waits alongside aggregate output. A service can complete more total work while an individual agent still waits too long to be useful.

FlashStates vs vLLM, SGLang and local exllamav3

Stock vLLM is the directly measured baseline in Kundu's H200 comparison; SGLang and exllamav3 answer different adoption questions. There is no single result here that ranks every serving engine.

Alternative Why you might choose it What changes the decision
Stock vLLM 0.33.0 You want the baseline without this research overlay Kundu reports 6.7% to 26.9% better whole-run throughput with FlashStates, alongside failed parity checks
SGLang You already operate SGLang and want to follow its native hybrid-cache work Issue #4 reports merged state-cost eviction and checkpoint-level controls, with unit eviction under review
Marconi You are comparing selective hybrid-state retention designs The alphaXiv overview distinguishes Marconi's tunable priority coefficient from FlashStates' policy, which still fits prefill costs per model
HiState You are investigating memory tiers and adaptive checkpoint intervals The overview reports no head-to-head comparison, so it supplies no winner
exllamav3 with Auryn's patches You run local inference and can evaluate CPU/disk cache behavior The adaptation has no Mooncake connector or dedicated mid-prefill export kernel

My preference for an existing deployment is to investigate the mechanism in the engine you already own before taking on a fork. The independent comparison gives vLLM operators a concrete reason to evaluate FlashStates. SGLang operators should track the individual merged features, rather than treating the whole design as either present or absent.

Source setup needs a GPU plan, not just a Python environment

The source guide calls for Linux x86_64, CUDA 13 drivers, Python 3.12 and uv. It separates the vLLM overlay, optional Mooncake store and replay client, so a small successful subproject install doesn't describe the full serving setup.

The README's ready-made images target B200 and GB300 configurations. Its serving example mounts model files read-only, enables prefix caching and supplies explicit checkpoint, eviction and model-cost settings. Clients retain the vLLM OpenAI-compatible API. You still need the model files and a configuration appropriate to the hardware.

Mooncake is an optional host-memory tier, not a prerequisite for Kundu's GPU-only results. Choosing it introduces a separate store configuration. The source guide also keeps the official baseline in a separate environment, which matters when your question is whether FlashStates improves on stock vLLM rather than on another mixture of patches.

Auryn's local implementation adds another practical lesson: dynamic content early in a prompt breaks exact-prefix reuse. A changing date or session identifier before a large static block can make that block expensive again. Its persisted weights also differ from persisted model tensors: warming after restart replays stored token chains and pays prefill work again.

Security defaults: the example uses host networking

The README's Docker command uses --network host and --ipc=host, alongside a read-only model mount. Those example settings deserve an infrastructure review before you expose the server to clients.

Treat the command as a model-serving example, then decide where your gateway, access controls and network boundaries belong. A read-only model mount controls writes to that directory; it doesn't answer who can send requests to the service. The failed Compose attempt's localhost-binding summary is no evidence of the published Docker command's default exposure.

What the other reviews get wrong

AINEWS Radar's account misses Kundu's independent H200 comparison, while Satyajit's older analysis predates an upstream vLLM decode-checkpoint change. Both distinctions affect what you should conclude today.

AINEWS Radar says its account contains no independent benchmarks or reproduction. Kundu's experiment supplies a controlled independent comparison, but it doesn't reproduce the authors' B200 headline: the GPU, model and workload differ.

Satyajit's November 9 analysis describes vLLM as never saving recurrent checkpoints during decode. The November 19 vLLM RFC says main already retains the latest block boundary crossed during decode. Retention at finer prefix-match boundaries is the proposal, not the entire idea of retaining decode state.

Two project statements also need narrower readings. The landing page says outputs stay bit-identical. Kundu's failed exact-output comparison limits that as a deployment-wide promise, without isolating checkpointing as the cause or disproving the narrower kernel claim. The install guide's options-off equivalence has exceptions: the fork commit documents three always-on fixes, involving unwritten Mamba state columns, the GLM MTP layer and overlaid KV copies.

Kirill Pokidov's LinkedIn summary goes too far in saying redundant prefill is eliminated. Kundu still measured 20.09 million later-turn uncached tokens under pressure. FlashStates reduces repeated work; it doesn't make cache capacity unlimited.

Finally, Satyajit's analysis separates serving measurements from the pricing overlay behind profit and user-cost claims. Faster token serving isn't by itself a measured reduction in your bill or an increase in your margin.

Questions people ask

SGLang support is in progress: issue #4 lists merged components and unit eviction under review, rather than a completed supported FlashStates edition for SGLang. The maintainer's reply is the useful starting point.

Is there a supported SGLang version of FlashStates that I can test?

The maintainer reports --radix-eviction-policy state_cost and --hybrid-cache-checkpoints-level merged into main. Unit eviction, --enable-mamba-unit-eviction, is under review and off by default in that proposal. Follow the roadmap linked from issue #4 for the individual pieces rather than assuming a full supported port.

How much of the reported 3× gain comes from dependency-aware retention versus exact-turn checkpoints?

There is no defensible universal split. The Qwen ablation rises from 24.1K output tokens per minute per GPU for vLLM to 34.2K with checkpointing, 36.8K with admission and 38.8K with the complete system at 48 sessions. Those model-specific staged results don't allocate the separate GLM headline. Kundu enabled four features together, so his experiment doesn't isolate their individual contributions.

Does FlashStates retain its advantage with shorter sessions?

The shorter-session question is open in Colin-Mario Köse's comment on Sudip Roy's post. Kundu's four-session result demonstrates a benefit when sessions fit, but changing concurrency isn't a controlled session-length experiment. His fixed eight-turn design gives no general short-session answer.

What happens to throughput between 40 and 48 concurrent agents?

The discussion asks for the curve between those loads; it doesn't measure the intervening behavior. Kundu compares four and 123 sessions and explicitly says a session-count sweep is needed. Don't interpolate an operational limit from those two settings.

At what load does throughput start falling, and how much cache headroom is needed?

There is no portable agent-count or cache-headroom rule. The authors' profiling shows capacity misses growing from about 40 sessions in their configuration, while Kundu says his overloaded setting cannot locate peak capacity. The project's capacity discussion belongs to its coding traces and equal-memory comparison, not every GPU.

Should upstream vLLM expose a separate decode checkpoint-retention interval, and what should it be called?

The November 19 RFC proposes a separate decode interval, with prefix_cache_decode_retention_interval as an example name still to be agreed. It distinguishes default retention at the last prefix-match boundary, periodic states with a positive interval and compatibility behavior with None. That is a design proposal seeking feedback, not a settled API recommendation.

Should the exllamav3 adaptation derive GDS coefficients per model rather than require manual configuration?

Auryn explicitly asks reviewers that question; his article doesn't resolve it. GDS, GreedyDual-Size, ranks retained checkpoints by saved processing cost relative to storage. The adaptation exposes manually configured coefficients, so automatic derivation is a review question rather than a shipped feature.

What we have not tested yet

Our successful subproject setup checks and stopped deployment attempt leave GPU inference, model loading, answer quality and production reliability untested by us. We did not reproduce the B200 headline, Kundu's H200 experiment or Auryn's local adaptation.

Kundu's random-token workload evaluates serving behavior, not coding ability or task completion. Its exact-output match rates were 71.43% and 80.92%, both below the predefined 98% bar. Some first-turn outputs already differed, before checkpoint reuse could explain them. The author lists numerical sensitivity, batching, scheduling and resume behavior as possible causes, and reports warnings about uncalibrated FP8 KV scaling factors. These are unresolved causes, not a diagnosis or a measured quality loss. Once outputs diverged, subsequent prompt contents differed too, although sequence lengths and workload shape stayed matched.

We have no security audit, load sweep on the intended deployment, restart-recovery exercise or evidence of broad customer adoption. The upstream RFC also leaves pipeline parallelism and some offload connectors outside its initial coverage. A useful next evaluation pairs a load sweep with output checks on representative tasks, keeping model revision, cache budget and engine version fixed. Decide from the first-token waits and completed correct tasks your users actually receive.

Alternatives

ProjectWhat it isPick it when
vLLM gh↗The stock 0.33.0 baseline in Kundu's controlled H200 comparison.Choose the stock baseline when you want to avoid this research overlay, then compare your workload before taking on the fork.
SGLang gh↗An alternative serving engine adopting related hybrid-cache mechanisms feature by feature.Stay with your SGLang deployment when you prefer to evaluate native merged controls and follow the unit-eviction proposal.
exllamav3Auryn reports a separate FlashStates-inspired implementation for local inference with CPU and disk tiers.Consider that adaptation if your target is local inference and you can evaluate the patches, rather than adopting the vLLM and Mooncake stack.

What people are saying

  1. [velocity-scout] FutureMLS-Lab/FlashStates

Sources

  1. FlashStates/README.md at main · FutureMLS-Lab/FlashStates
  2. FlashStates - Hybrid Model Inference
  3. raw.githubusercontent.com/FutureMLS-Lab/FlashStates/HEAD/docs/install.md
  4. [RFC]: Decode states at prefix-match boundaries for hybrid prefix caching (FlashStates)
  5. Auryn on X: "The other half of the cache" / X
  6. FlashStates: Unlocking Flexible Hybrid State Management for Agentic LLM Inference
  7. sglang适配版本
  8. Flash States

More llm tools reviews