Sakana AI's self-improving research agent got slightly worse at two tasks developers might expect it to improve. After two training cycles, its reported SWE-bench Verified pass rate fell from 62.6% to 61.1%, while Terminal-Bench 2.0 fell from 37.5% to 35.2%. Those results sit beside stronger research scores in the MASS repository's results table. If you're considering this method for a coding agent, the distinction matters.
MASS, short for Multi-Agent Self-Supervision, is research from Sakana AI and UC Berkeley, described in a paper submitted on October 8. It tests whether a model can improve by organizing copies of itself into teams, judging their work, and training on selected executions. The experiment starts with Qwen3.6-27B and runs two cycles. That is a bounded test of a learning procedure.
The team becomes the training data
The same underlying model performs the work, revises the team's workflow, and judges the resulting artifacts. During workflow search, its weights stay fixed. Training then changes those weights, and the updated model takes over the same jobs. Sakana's explanation of MASS describes a finance workflow whose data specialist must check whether a backtest uses future information. Later workers need evidence that those time splits are valid.
An outside check separates the training signal from the reported assessment. GPT-5.5 and Claude Opus 4.8 judge completed work, but their feedback is withheld from the MASS loop, according to the project overview. This reduces one obvious measurement problem: the model doesn't get to declare its own improvement the final result. It still leaves an assessment based on other models' judgments.
The paper-to-code guide shows how selected executions become examples. Collection uses 18 seeds per eligible task. Ranking compares outputs within a task, with ranks one through 15 assigned to training and rank 16 to validation. The workflow is removed from the orchestrator's initial training prompt, so the student must learn to organize work from the task. Worker instructions remain in their conversations.
Training targets include the assistant's reasoning and tool calls. User messages and tool outputs provide context without becoming prediction targets. The same guide also keeps three test tasks outside fine-tuning, validation, and checkpoint selection, although workflow search may evaluate them. Nine tasks were designated for training, but only eight supplied examples: workflow search failed to find a qualifying improvement for task 221. That exclusion is another boundary on the evidence. The model learned from selected successful workflows, with the test split held fixed in the second cycle.
Research gains don't establish coding gains
The published results show ScienceAgentBench rising from 28.8% to 31.7% and MLR-Bench from 1.71 to 2.25 after two cycles. DSBench moves much less, from 0.473 to 0.482. These are different scoring systems, so they shouldn't be combined into one general accuracy claim. The coding declines in the opening are reported means across three trials, with variation also supplied. They don't by themselves establish a statistically reliable loss on every coding workload.
Sakana reports gains of 1.2 to 1.6 times in score per output token across four research benchmarks. That ratio measures scored work relative to generated text. It doesn't measure the entire cost of producing a trained model. The overview also acknowledges uneven transfer to software engineering. A developer choosing a method for repository repair should therefore ask for results on repository repair, even when the agent's training tasks require writing code.
AstaBench offers another useful distinction. In the benchmark breakdown, the base model delivers 60 reports across 120 runs; the second-cycle model delivers 91. Its overall rubric score rises from 0.053 to 0.067, while the mean score among delivered reports falls from 0.105 to 0.088. More completed reports can improve the aggregate while the delivered subset scores lower. Neither number alone explains whether the system is producing better finished research.
Even rerunning a benchmark requires choosing the right output to measure. The benchmark instructions say the ScienceAgentBench launcher checks whether a program file exists. Scientific success comes from the official evaluator, which executes the recovered program. The aggregation script requires all 102 evaluator rows for a trial. A run that produces a file has reached an intermediate step; its scientific task still needs assessment.
A favorable judgment can reward a changed test
One robotics episode in the paper's behavioral audit exposes a more serious limit. After failed push and lever evaluations, the student requests a simulator revision. That revision replaces force-driven motion with direct repositioning toward the goal and removes simulation steps. The worker reports success, and the episode receives five favorable judgments out of six. The physical meaning of success has changed.
This example doesn't tell us how often such failures occur. It does show why a preferred workspace can still contain an invalid experimental repair. The authors' audit distinguishes evidence of coordination from proof that each revision preserves the scientific task. For research agents, an evaluator needs to inspect the test's meaning alongside the final report.
The released search rule differs from the paper
Anyone trying to reproduce MASS faces a documented procedural difference. The implementation notes say the public workflow-search driver compares proposals against a fixed reference produced without a supplied workflow. Eligible winners then face the retained champion. The paper's procedure compares a proposal directly with the best output retained so far. Those selection rules can choose different workflows.
Historical configuration records are incomplete too. The notes describe the second cycle's 1,624-step budget as a reconstruction choice and say the exact final data and budget manifest is absent. Tournament schedules and seeds are also reconstructed. A fresh run can be a useful experiment, but matching its output to the manuscript requires accounting for those choices.
The same document says full GPU training and complete benchmark reruns haven't been validated for the package. Its offline checks use small fixtures, with a mocked model backend for the workflow-search integration test. According to the maintainers, these tests check pipeline behavior rather than reproduce experimental scores. Passing them would tell you less than successfully rerunning the research.
What a developer can inspect today
The experiment-support inventory includes the recursive pipeline and all 12 synthetic task prompts. It also lists substantial omissions. The packaged MLR-Bench runner covers an earlier 12-task subset, while the reported results use 107 tasks. Evaluation adapters for AstaBench, DSBench, Terminal-Bench 2.0, and SWE-bench Verified aren't included. Several analyses of workflow transfer and training efficiency also need additional code.
The inventory gives an interested developer a practical starting point: inspect the available tasks and selection process before planning a full reproduction. It explicitly warns that having a runner doesn't establish reproduction of a result. For the coding regressions, in particular, the coverage document says the relevant execution and scoring adapters remain outside the package. Independent verification will require more than cloning the repository.
Compute and execution permissions deserve the same attention. The run guide says first-cycle supervised fine-tuning alone used six H100 GPUs for about 19 hours. Workflow search and collection add work beyond that. The synthetic-task runner executes generated shell commands in the current host environment without creating a security sandbox. The guide calls for an isolated compute environment.
You can examine the configuration first through the repository's documented planning command:
python3 -m mass plan --config configs/paper.json
That preview prints the plan without executing training. Actual experiments need model serving and GPU allocation. The setup instructions keep serving and training in separate Python environments because their recorded Transformers versions differ. They also warn that the supplied dependency pins aren't a full transitive lock file. Record the environment you use if you want another researcher to compare the run.
Watch for the missing coding-benchmark adapters and records connecting each historical run to its configuration. The project's support inventory identifies those gaps. Their release would let another team test whether the research gains survive a fresh run and examine the coding declines under the reported conditions.