Passing tests leave agent quality unanswered
Harness² passed 70 pytest tests in our sandbox. Its agent-quality gains remain unverified by our lab. The adoption question is whether you can run a controlled comparison that establishes what changing the agent's instructions and tools achieves.
The project README describes a framework that keeps model weights fixed while changing the surrounding harness. Candidate harnesses attempt the same task, and an editor composes a refined harness from differences between executions. A persistent improvement harness retains adopted changes and lessons for later tasks.
This workflow suits researchers studying agent behavior with control over model spending and evaluation conditions. It is a poor fit for someone seeking a supported assistant service: the README says this is not an officially supported Google product.
What happened when we ran it
Our November 6, 2026 run measured commit 6868817. Installation succeeded in 15 seconds, the build succeeded in 1 second and pytest completed in 71 seconds: 70 passed, zero failed. The record lists 36 dependencies occupying 38 MB.
The configured sandbox had 3 CPUs and 8 GB RAM, with image label ghcr.io/astral-sh/uv:python3.12-bookworm. That label does not establish the actual Python or operating-system version. The checkout contained 140 files, approximately 19,477 source lines and occupied 8.1 MB.
Pip-audit reported zero known vulnerabilities. The audit record lists 35 scanned packages and one skipped package, so the result does not cover every recorded dependency. It also does not assess executor permissions. The repository scan found a tests directory, zero CI workflow files and no Dockerfile; those observations do not establish whether upstream benchmark containers are available.
These measurements establish successful repository checks for this checkout. They do not establish model-backed benchmark completion, output quality, API spending or gains over an unchanged agent. The README reports benchmark improvements, but those remain upstream results.
The demo and benchmarks need different preparation
The documented local demo uses deterministic editors and real Python executions without model credentials. It offers a way to inspect recursion mechanics. The supplied lab record does not establish that this demo ran.
The setup guide recommends Linux and Python 3.12 for benchmark runs. Model-backed work requires Node.js 20 or newer, Google Cloud CLI and authorized model access. LAB also needs Pandoc and Poppler; WorkBuddy-Bench needs Docker. Benchmark environments are separate, and the guide says to keep the main Harness² environment active for its commands.
The installation checker checks dependencies, credentials and configuration without calling a model. These checks do not test model availability. Access is tested when each model is first called, so a successful check does not establish that the first model request will succeed.
Network access can compromise the comparison
The experiment guide reports that a Codex rollout on JobBench downloaded its task's rubric without being instructed to. That upstream observation gives a specific reason to restrict access to grading material before evaluating results.
Executor network access is unrestricted, and Codex runs with its sandbox disabled. Upstream recommends network restrictions after setup to reduce access to benchmark data and grading criteria, and asks researchers to report the restrictions alongside results. Setup itself needs network access to download dependencies and data.
If your organization cannot enforce evaluation restrictions, postpone score comparisons. Passing repository tests does not establish that benchmark grading material was inaccessible to the executor.
Record the comparison conditions
The experiment guide distinguishes parallel candidate exploration from sequential refinement. Model-backed runs incur API costs for executor rollouts and improvement stages. Start with a named domain and a small task selection, and use the documented dry run to inspect the selection without model calls. The task cap applies per domain, rather than across the whole run.
Use the same benchmark, executor, subset, recursion budget, mode and seed for refinement, judging and reporting. When comparing budgets or modes, keep task selection, executor, improver, judge and web access fixed. A seed shuffles task order but does not control model sampling. JobBench data is resolved at download time; report the recorded dataset revision alongside results.
The supplied GitHub metadata records a last push of November 2, 2026. The supplied metadata does not establish whether a latest release exists. It also does not establish support response times or maintainer capacity.
Consider DSPy and GEPA for adjacent needs
DSPy supports modular AI systems and provides algorithms for optimizing prompts and weights. Consider it when your deliverable is a compositional Python language-model program.
GEPA optimizes systems with textual parameters against an evaluation metric. Consider it when you can provide an evaluator and want to search over text artifacts. These are workflow alternatives; the supplied measurements do not compare their performance with Harness².
Our editorial scale uses 1 for a documented blocker, 2 for substantial friction, 3 for a usable path with material limits, 4 for detailed support within the evaluated scope and 5 for broad validation across the claimed scope. Setup and maturity receive 3; documentation receives 4 for its coverage of prerequisites and evaluation controls. Community support remains unassessed. These judgments do not score agent quality or production adoption.
Before expanding an experiment, record an unchanged-agent baseline under the same task selection and network policy. Use the experiment guide's baseline-evaluation instructions to distinguish reference results from refined-harness results.