Caveman 4's advertised 54% cost cut comes mostly from lowering model reasoning effort on easier requests. If you already run your coding agent at medium effort, the project hasn't measured the saving against your setup. That qualification sits inside its own routing benchmark.
Released on 13 October, version 4.0 adds automatic model routing and reusable scripts to a toolkit best known for making AI agents answer tersely. Those additions give developers more ways to reduce a session's cost, with different evidence behind each one.
MrKeyoor's 16 October GitHub Trending snapshot recorded 943 stars gained that day. The repository's description still promotes a 65% token reduction through caveman-style speech. Its current README describes a much wider product, including input compression and model selection.
Auto mostly turns down reasoning effort
The interim run, dated 3 October, covers 85 of 169 planned paired tasks, with one attempt per task in each configuration. Auto's Claude-only pool averaged $3.59 per attempt against $7.83 for Opus 5.5 fixed at xhigh effort. The reported task scores were 43.7% and 44.3%, respectively. Costs use catalog list prices, rather than billed amounts.
Auto used Opus at medium effort for 87% of requests. Its overall score difference has a 95% interval spanning roughly seven points below the baseline to six above it. The comparison therefore leaves room for a meaningful quality difference. Raw trajectories aren't published, and today's router hasn't been remeasured against that build.
With Auto selected, the local proxy asks Caveman Cloud to choose a model and effort from configured providers. The release notes describe fallback to unrouted requests when a key is refused or a billing limit is reached. A signed-in runtime sends event batches to Cloud and respects a telemetry opt-out. These are operational details to review before putting another service in your agent's request path.
Shorter replies leave much of a coding session untouched
JetBrains tested the earlier response skill on real agent tasks in July. Its setup used Claude Code with Sonnet 5 at low reasoning effort, comparing stock behavior against forced Caveman activation. Across 82 paired tasks in the full run, output fell from about 592,000 to 542,000 tokens, an 8.5% reduction. Code and tool calls dominated the output, and the skill deliberately preserved them.
The researchers detected no statistically significant quality difference. They also found that fewer output tokens didn't guarantee a smaller total bill: one dependency-audit trial crossed a long-context pricing threshold and reversed the aggregate cost comparison. JetBrains attributed that outlier to task variability. Its experiment tests an earlier style skill, so it doesn't validate or disprove version 4's routing and compression results.
Caveman's own comparison against a simpler instruction, Answer concisely., narrows the claim further. Over ten developer questions, the standard skill reduced median output by another 3%, while the more aggressive ultracave mode reduced it by 35%. That was a single run on version 3.1.0 rules. Output length alone doesn't establish that every technical detail survived.
The same report says the response skill doesn't compress model thinking tokens or input context. It can add input overhead through its instructions. That makes an existing terse prompt a useful control: compare the tool against the behavior you already use, then include its added instructions and cache usage in the accounting. A shorter visible answer is only one part of that comparison.
Input compression helped CSV and made HTML worse
Caveman's separate tool-output benchmark reports a 33.2% reduction in provider-reported input tokens across 18 direct-versus-Caveman pairs. All 18 Caveman runs passed the exact-answer checks. The tasks used fixed fixtures such as logs and configuration data, with each arm reading a fixture once and returning a structured answer. This tests finding specified information in large outputs.
CSV workloads cut session input by 55.1%, while HTML used 9.9% more because no compression transform applied and the skill's overhead remained. The report counted cache tokens without weighting their different prices. Its 33.2% input reduction therefore isn't a measured 33.2% discount on a bill.
The session tests ran in August on an older engine. October measurements of compressed files use newer code, and the complete session benchmark hasn't been rerun on it. The repository publishes the report and provenance hashes without the raw harness or run artifacts needed to reproduce the result independently.
Compressing a prompt can make the answer more expensive
In the June CAVEWOMAN preprint, Morayo Danielle Adeyemi, Ryan A. Rossi and Franck Dernoncourt evaluated eight models across five datasets, separately compressing user prompts and model responses. Terse output reduced realized cost on most API models, with reported per-model cost reductions of 1.4 to 2.4 times.
Compressed user prompts produced the opposite result in those experiments. Models compensated with longer responses as accuracy fell, raising average cost across the five benchmarks to about 1.15 times the uncompressed baseline. The study also found cases where a compressed answer remained correct under the task's scoring while its wording no longer conveyed all the information in the model's unconstrained reference answer. Passing a task score and preserving the full answer are different checks.
Those findings concern linguistic compression under the paper's protocol. They don't establish how well Caveman's current proxy selects and recovers information from logs. They do explain why removing words deserves its own accuracy test, even when a token counter reports a large reduction. A result about terse responses cannot automatically answer a question about compressed instructions.
Reusable scripts need a fair comparison too
Version 4 also bundles Blocks, which keeps small reusable scripts in a repository instead of having an agent regenerate them. The separate Blocks project writes an index and rules into AGENTS.md, verifies each block's example, and can recheck it in CI. Its shell hook suggests an existing block when it recognizes a matching script, without denying or rewriting the command.
The project's examples compare Blocks with shortcuts agents already use. For a 326-test Go run, its summary returned 76 tokens against 397 from tail -20. A JSON inspection barely changed, from 236 tokens to 227. Its definition extractor lost to ordinary grep, returning 489 tokens against 250. The project hasn't measured how often agents follow the hints or the effect across whole sessions.
Our Caveman review describes an August checkout whose dependencies installed successfully but whose test command stopped on a Node option rejected by the runtime. That run predates version 4 and should be read as evidence about that checkout.
The current README lets you inspect enabled modules with caveman status and switch them separately. The response skill is also available without the proxy. That separation makes a useful trial possible: start with the component addressing your measured waste, rather than attributing every change in a combined install to the caveman voice.
For developers evaluating it now, the project's measurement guidance is concrete: run the same task with and without Caveman and compare provider usage or billing. Its session statistics report savings as unknown when there is no measured comparison. Keep a correctness check alongside that comparison, since the least expensive run is little use if it leaves the task unfinished.
The next evidence to watch is a completed routing run, followed by a comparison against medium-effort settings and published traces. Version 4 makes model selection part of Caveman's job. Until that testing catches up, judge Auto against your usual agent settings before treating the advertised discount as a budget forecast.