A fixed panel can detect drift in one serving path
A disappointing coding session cannot tell you whether a provider changed its model. LiveNERF gives that suspicion a narrower question: do answers to the same fixed questions change across repeated observations? The project runs through a pinned Claude Code client, freezes prompts and graders, and retains raw evaluation logs. Its useful contribution is the discipline around a comparison. You still have to decide whether the comparison represents the work you care about.
The codebase we measured occupied 2.8 MB across 86 files, with roughly 4,895 source lines. That is a manageable amount of project code to inspect before entrusting it with a long study. Its main dependency is Inspect, which supplies evaluation machinery and a model-provider interface. LiveNERF adds its own Claude Code wrapper, panel selection, scheduling, and statistical decision rules, rather than presenting itself as a replacement for a general evaluation framework.
What happened when we ran it
Our run installed 113 Python packages in 33 seconds, occupying 620 MB, then built successfully in 2 seconds. The October 16, 2026 sandbox used commit 2fa3eb6 in the uv Python 3.12 Bookworm image, with 3 CPUs, 8 GB of RAM, and no secrets. Pip-audit reported 0 known vulnerabilities. The package path worked, though its installed footprint was much larger than the checkout.
The test step failed with exit 1 after 24 seconds. Pytest reported 67 passed and 6 failed out of 73, and every listed failure said async def functions were not natively supported. Those were provider checks covering parsing, explicit effort, CLI errors, classifier fallbacks and retries, and injected context. The log does not establish a provider defect or tell us that adding a particular plugin would make the suite pass. It leaves those paths unverified in our environment.
Freezing Claude Code is part of the measurement
The setup instructions require a logged-in Claude Code installation, uv, and Python. After dependency setup, you record the CLI version and preserve a copy of its binary away from automatic updates. The runner refuses a version mismatch. That is a sensible restriction: if the wrapper changes while the model is under observation, a shift in results can come from either side of the request. A frozen binary removes one moving part.
Our 33-second installation did not include collection through an authenticated account. The documented workflow adds calibration, panel design, locking, validation, and a preflight probe before scheduling begins. Budget guards use the account's usage meters and may postpone a daily attempt. You need enough allowance to keep the series running, plus durable storage for the raw .eval logs. Since those logs are gitignored, cloning the repository alone does not preserve your evidence.
Tool-free questions leave coding complaints unresolved
LiveNERF intentionally removes project files, tools, MCP servers, hooks, and memory from a measured call. Each request uses a frozen prompt in an empty working directory, with effort passed explicitly. This helps isolate answer behavior. It also excludes much of what makes a coding assistant useful: reading a repository, choosing tools, repairing a failed command, and carrying context across a task. Those are different evaluation requirements.
The 6 unsupported async tests matter here because isolation and response classification live near the provider boundary. A research harness needs confidence that it rejects injected context and does not score a fallback response as the intended model. Separately, issue #12 on software-engineering coverage asks whether the panel reflects coding complaints. That issue concerns task selection, not our pytest failure. Both deserve attention before extending the findings to your daily development work.
A null result has limits stated in the README
The current protocol compares a launch-week baseline with later windows and uses a control arm to help detect shared platform changes. The README also states that a same-family model swap was not distinguishable in its validation sample. That admission is useful. A result showing no detected change cannot establish that the provider kept identical weights, routing, latency, or behavior on tasks absent from the panel. The instrument has a stated sensitivity and a defined subject.
Our scan found 1 CI workflow file, a tests directory, and no Dockerfile. GitHub showed a push on October 16, 2026, with 3 open issues and PRs and no latest release returned by the API. Recent discussions address task coverage and speed tracking. Read together, those signals describe an active, small study repository. The absence of a release tag is less informative than whether the protocol, data, and analysis remain inspectable.
Reproduce the checks before collecting a new series
Use LiveNERF when you want to maintain its specific kind of evidence and are prepared to defend exclusions and baseline choices. Inspect is the more direct choice for a custom study, while promptfoo suits application prompt regressions. The lm-evaluation-harness is another starting point for comparisons across benchmark tasks. Those alternatives change the question as well as the tooling, so choose according to the claim you eventually need to support.
The 620 MB environment and failed provider checks are concrete adoption work, even though the protocol documentation is unusually candid. Before scheduling a fresh series, resolve the async runner failure, confirm the pinned client and isolation checks, and retain the raw logs. Then phrase any result around the serving path and tasks actually observed. A careful time series earns trust by keeping its conclusion inside those boundaries.