It rewrites code to improve one number
pi-autoresearch adds an experiment loop to the pi terminal coding agent. You give it a command, metric, direction, and file scope. The agent edits code, commits the candidate, runs the command, records the result, then keeps an improvement or reverts a regression. Suitable targets include test duration, bundle size, build time, model loss, and Lighthouse scores.
The project is small enough to inspect before granting that authority. Our checkout contained 39 files, about 6,822 lines of source, and occupied 0.5 MB. The extension supplies the experiment tools, terminal dashboard, and automatic continuation. Separate skills create the session, add optional hooks, and turn successful work into independent branches.
The .auto folder leaves a usable experiment record
Each session keeps its working state under .auto. The prompt file records the objective, scope, tried ideas, dead ends, and useful findings. The measurement script defines the benchmark, while an append-only JSONL file records every run. After pi compacts its conversation or a process restarts, the agent rereads those files and Git history. That is a practical answer to the memory loss that makes long coding-agent jobs hard to audit.
Version 1.6.2 raised automatic continuation to 200 turns and added a stop after more than 20 consecutive discards or crashes in a segment. The 0.5 MB checkout also includes a finalization skill that proposes groups of successful changes and waits for approval. It creates each approved group from the merge base, and groups cannot share files. That rule limits how commits can be combined, but it makes the resulting branches easier to review separately.
What happened when we ran it
Our sandbox installed 127 pnpm packages in 35 seconds, using 174 MB on disk. We tested commit 00062fb, which was also the current main commit when we fetched the project. The environment was a fresh unprivileged Debian container with Node 22, 3 CPUs, 8 GB of RAM, and no secrets. Installation completed without an error.
The repository has no build script or target, so there was nothing to compile and we skipped that step. Node's test runner completed in 5 seconds with 46 passes and 0 failures out of 46. npm audit reported 0 known vulnerabilities at every listed severity. We also found 2 CI workflow files and a tests directory, but no Dockerfile. Those results cover the extension itself, not the projects or model providers used during an optimization session.
One clean test run cannot make a noisy metric trustworthy
The confidence display compares the best improvement with a noise estimate after at least 3 experiments. It can flag a result that sits within ordinary variation, yet it never rejects that result automatically. Open issue 69 asks for repeated sampling and controlled A/B replay because the current loop has no first-class mechanism for either. Timing jitter, cache state, stochastic training, or changing background load can therefore turn one sample into a false winner.
All 46 extension tests passed in our run, but they cannot validate a measurement script written for another repository. Put repetitions and aggregation inside .auto/measure.sh when the target is noisy. Better still, use a dedicated optimizer such as Optuna when the problem can be expressed as a parameter search with recorded trials. pi-autoresearch earns its keep when useful experiments require source edits that a parameter sampler cannot describe.
Bounded scores can trap the loop after success
Open issue 64 documents a session whose binary feature score rose from 0 to 1 and passed its checks. Later confirmation runs also scored 1, which meant they could not count as improvements. The loop recorded those successful confirmations as discards and continued asking for more iterations. The proposed completion and confirmation states are still ideas in an open issue, so users should not describe them as shipped behavior.
The clean 5-second test result does not settle that workflow gap. For pass/fail goals, zero-error targets, or scores with a hard ceiling, stop the session when the objective is reached or choose a metric with room to move. An endless instruction makes sense for open-ended optimization. It is a poor match for a task that has a clear terminal state but no way to express one to the extension.
Correctness and cost limits are operator settings
A .auto/checks.sh file can run tests, types, or lint after a benchmark improves. A failed check blocks the keep and records a separate status. The file is optional, which means the default loop can preserve a faster change that breaks behavior outside the primary metric. Before the first unattended iteration, add checks that cover the behavior you refuse to trade away and keep the work on a dedicated branch.
The initial 127-package install is predictable; model usage is not. The README warns that an autonomous loop can consume substantial tokens. maxIterations caps experiments within the session, while provider-side budgets create an independent ceiling. Hooks add more reach because they can run external searches, write a journal, or send notifications. Review those scripts as carefully as the measurement command since they execute around each iteration.
July code and August issue activity point to a maintained project
GitHub showed 7,869 stars and 15 combined open issues and pull requests when fetched. The last code push was July 15, 2026, six days after the v1.6.2 release. Issue and pull-request activity continued through August 25. That combination is healthier evidence than the release date alone, though several proposed fixes and larger features remain unmerged.
Issue 86 is a current compatibility blemish: pi 0.84.2 added transcript search on Ctrl+Shift+F, the same default used by pi-autoresearch's full-screen dashboard. Users report a startup warning and loss of pi's built-in binding. The README documents a configuration file that can remap or disable the extension shortcut. For a 39-file project, that workaround is manageable, but a fresh install should not require discovering a conflict report.