Passing project tests leaves the live repair workflow untested
RepoPilot's appeal is its documented separation of proposed changes from permission to publish them. Codex proposes tests and repairs; the controller requires runner evidence before publishing a separate branch and draft PR. Maintainers decide whether to merge.
For a public repository with reliable tests, that offers a concrete workflow to evaluate. Our recorded checks establish a narrower result: dependency installation, compilation and the project's test command completed successfully. They do not establish safe autonomous code delivery.
What our recorded run establishes
Our run began on October 5, 2026 at 05:51:52.225 UTC and checked commit 6a5597d6c8e2bc4d61079bd7201f59ce7f86c9ce. The build output identified repopilot-agent@1.5.0. The sandbox record specifies 3 CPUs, 8 GB of memory and image lab-node:22. That image label does not establish the actual Node runtime version.
Dependency installation exited successfully after 55.674 seconds, rounded to 56 seconds in the summary. The TypeScript build exited successfully after 13.909 seconds, rounded to 14. The test command exited successfully after 226.103 seconds, rounded to 226. Its retained output reported 180 passed, 0 failed, 0 cancelled and 0 skipped. We have not reconciled complete logs to establish exhaustive unique test coverage.
The stored dependency metrics record count 21 and 400 MB. The raw installation tail instead says 12 packages were added. Neither figure should be presented as a reconciled installed-package count. The npm audit record contains zero advisories across its severity fields. Coverage is unknown; this is not evidence that every dependency or the product is secure.
The checkout scan recorded 138 files, approximately 7,504 source lines, 0.6 MB, two CI workflow files and a tests directory. It did not detect a default Dockerfile. The README separately provides Dockerfile.agent for the agent image.
The pinned README says the project tests use mocked APIs, agents and runners alongside synthetic local Git/Node fixtures. They do not use model credits, launch Docker or write GitHub content. Our run therefore does not verify a live model call, target-repository workload, Docker repair, installed CLI binary or published PR.
What the verification design promises
According to the README, RepoPilot pins revisions and trusted base-branch policy, freezes generated tests before repair and checks case identities across executions. A base-pass/head-fail result can identify a regression. A base-fail/head-pass result qualifies as new behavior only with supporting requirement evidence. Ambiguous failures, missing cases and execution errors require review.
The architecture document adds an important limit: repeated failure identities and fingerprints detect changing failures, but do not provide statistical proof that a test is stable. Generated tests also cannot establish that a proposed behavior matches the maintainer's intent. Human review remains necessary.
Trusted .repopilot/policy.json rules can check literal text and direct or qualified calls. Nested AGENTS.md files inform semantic review. The documented AST rules do not resolve aliases or perform whole-program analysis, and prose conflicts require human interpretation.
Keep the measured checkout separate from current documentation
The recorded build identifies v1.5.0. Its release notes describe optional Planner, Tester, Developer and Reviewer Codex roles, shared call/token accounting, durable handoffs, post-merge tracking and stored-evidence evaluation. Portable archives include Node.js and dependencies, but require external Git and Docker. The packages are not code-signed or notarized.
The supplied current README describes v1.8.0. It includes optional bounded parallel execution of independent task-graph nodes, a persistent service with signed Webhooks and polling, and a local read-only console. Those are documented current capabilities, not results from our v1.5.0 checkout run. A blanket claim that RepoPilot has no Webhooks or dashboard would therefore be misleading. The current documentation still excludes fork execution, browser E2E, automatic dependency installation, hosted administration and automatic merging.
The worker needs its own boundary
The security guide recommends a dedicated development machine or VM without production credentials. Test containers have no network by default; configured dependency services use a disposable internal Docker network. Containers share the worker kernel and do not provide a complete boundary against hostile code.
The agent receives an OpenAI key and repository text, while GitHub credentials remain with the controller. Self-hosting does not make inference offline. Local reports and snapshots can contain code and logs, and the preview does not claim comprehensive secret redaction. Test code shares a container with its runtime and reporter, so structured evidence is not tamper-proof attestation against malicious code.
How to decide
Start with a local check on a public text repository, using a trusted runner image and pre-provisioned dependencies. Keep publishing disabled while reviewing the resulting evidence. Model calls, Docker execution and GitHub publication each need separate validation in your environment.
Aider suits direct terminal pair programming. OpenHands Agent Canvas suits operators who want a control center spanning local, remote and cloud agent backends. RepoPilot's Codex SDK dependency is part of its implementation; an SDK alone is not a substitute for operating this controller.
Rating rubric and evidence limits
These ratings describe readiness for the supervised public-repository workflow above. A score of 1 means substantial uncertainty or missing evidence; 2 means a limited basis for evaluation; 3 means a usable starting point with material operator work; 4 means detailed support for this workflow with remaining gaps; 5 requires comprehensive evidence for the dimension. Unknowns lower confidence rather than establish product defects. Ratings are editorial judgments, not benchmarks or security certification.
Setup earns 3/5 because local commands succeeded but Docker and credentialed operation remain untested. Documentation earns 4/5 for specific setup, verification, recovery and security guidance, with version differences requiring care. Community earns 1/5 because the supplied evidence does not establish adoption or support responsiveness. Maturity earns 2/5 because packaged releases and local checks provide a basis for evaluation, while the project remains a developer preview and our run excludes live operation.

