Prime Intellect gave frontier AI models as long as eight days on nodes with eight H200 GPUs, then repeated the exercise until it had 153 autonomous runs. The expensive part of the result is also the easy part to miss: none of the agents produced a fundamentally new method. The best systems pulled ahead by measuring noise, revisiting weak results, and deleting improvements that stopped working. For developers deciding what sort of research can be handed to an agent today, that distinction matters more than the leaderboard.
The experiment, called NanoGPT Speedrun Frontier, tested 18 models on a narrow optimization task. Each agent received a small GPT training program, a rulebook, access to a GPU node, and one goal: reach a target validation loss in as few training steps as possible. Prime Intellect published the interactive results along with selected traces, scratchpads, and record submissions, making it possible to inspect more than the final scores.
The headline winner, Fable 5, cut the verified recipe from Prime Intellect's 3,290-step baseline to 2,726 steps. That closed 81.7% of the gap to a 2,600-step human record claim. Opus 5 reached 2,920 steps, while a Kimi K3 run reached 2,930. Prime Intellect says Fable and Opus also led when it compared runs at equal budgets measured in time, experiment count, or output tokens.
Those numbers do not establish a general ranking of AI models. They describe performance on one compact training problem under particular agent harnesses, reasoning settings, and compute budgets. Prime Intellect itself says the benchmark has substantial variance and that it lacks strong evidence that speedrun methods transfer to large model training. The more defensible finding is narrower: some models managed a long, noisy experimental process much better than others.
A benchmark built around noisy evidence
The task trains a 124 million parameter GPT and asks how few steps it needs to reach a validation loss of 3.28. The agent can edit the training recipe, mostly through optimizer choices such as preconditioning, learning-rate schedules, limits on weight and update magnitudes, and weight averaging near the end. It knows that a better recipe exists, but it has no internet access and must find improvements through experiments in its own workspace.
A single promising run cannot claim a record. The harness retrains a proposed recipe eight times on fixed seeds that the agent cannot change. A frozen verifier accepts the result only if the mean loss beats 3.27859. Prime Intellect says that margin puts the chance of passing through luck alone at roughly one in a thousand, close to the statistical rule used by the upstream speedrun.
That rule was necessary because earlier agents found bad ways through a loose evaluation. Some manipulated the number of samples or ended runs too soon. In the published experiment, the sandbox isolated each agent's files, exposed the dataset as read-only, and allowed outside traffic only to the model API through a logging proxy. An independent model monitored runs hourly until the team stopped that check after hundreds of reports found no cheating or sandbox escape.
The benchmark still contains noise that validation cannot erase. Two runs using the same model and harness landed about 54 steps apart after 24 agent-hours, according to Prime Intellect. The team launched at least three seeds for most configurations, selected the best after about 24 hours, and extended promising runs. That procedure makes the test affordable, but selecting the best seed can widen the distance between a model's best published run and its typical result.
The leaders treated experiments as evidence
Nearly every model rediscovered similar optimizer ideas. The separation appeared in what happened after an experiment returned a number. Weaker agents often rejected an entire family of changes after one seed, mistook their own implementation bugs for negative evidence, or discarded small gains that did not cross the record threshold alone. Stronger agents tested borderline results on multiple seeds and paid for the full eight-seed validation only when their noise estimates justified it.
The better runs also rechecked earlier conclusions after the recipe changed. Opus 5 returned to beta-2 tuning under a new configuration and found a record. Kimi K3 removed two mechanisms that had helped a previous recipe after a later normalization made them useless. Fable 5 eventually tested pairs of changes that were individually worse but worked together; Prime Intellect reports that one late recheck saved 31 training steps.
This behavior resembles careful engineering more than a flash of discovery. A negative benchmark result applies to the exact code and conditions tested. Once another part of the system changes, the old result may no longer hold. The leading agents kept enough experimental history to revisit those branches, while pruning changes that no longer earned their cost. That is a practical pattern for agent-assisted performance work: preserve measurements and make the agent revalidate the full stack after each merge.
Some agents built their own research tools along the way. A Kimi K3 run created functions to generate controlled optimizer variants, launch them, compare loss curves, and restore a clean baseline. Other traces show configuration builders and ablation generators replacing repeated source edits. The public repository includes sanitized event logs, subagent transcripts, and scratchpads for selected runs, so readers can examine how those workflows developed rather than taking a summary at face value.
The tools did not guarantee a good scientific decision. In one example, Kimi optimized coefficients for a whitening transformation that looked cleaner in a numerical simulation. The actual training run performed worse, with validation loss moving from 3.28684 to 3.28895. The useful action was abandoning the elegant surrogate result after the workload contradicted it. The benchmark rewards that willingness to update because only measured training performance counts.
The model score is entangled with the harness
Each leaderboard row names both a model and the software used to operate it, such as Claude Code, Codex, Kimi Code, or Prime Agent. That pairing limits simple model comparisons. Prime Agent, for example, gives a model a persistent IPython kernel in which it can develop reusable analysis functions. A model running through another harness may receive different affordances for parallel work, state, or recovery after an error.
Prime Intellect also changed parts of the launcher and monitoring setup while the runs were underway, including restart behavior, goal-completion detection, and one change affecting subagent spawning. The team says it did not observe a major effect and retained the healthiest run during multi-day tests. Still, the repository labels runs made under a serial rulebook and says those are being rerun under the standard rulebook. Comparisons across the table should retain that qualifier.
Compute and token totals vary sharply as well. The published table attributes 800 million total tokens and 8.7 days to the leading Fable 5 trajectory. The GPT-5.6 Sol run used 2.9 billion total tokens over 6.1 days and finished at 3,042 steps. Grok 4.5 reached 3,120 steps with 46 million total tokens over 2.7 days. Those are reported trajectory totals, not a price comparison, but they show why a bare rank is a poor procurement guide.
There is another boundary around novelty. The agents were cut off from the internet, yet their training data may include different papers because the models have different knowledge cutoffs. Prime Intellect deliberately accepted that mismatch after limited trials with paper search made agents slightly less creative. Even with that restriction, the team says the winning ingredients resembled methods already present in the literature.
What developers can take from the traces
The experiment supports a specific use of research agents: give them a bounded objective, executable measurements, immutable validation, and enough time to revise their own tooling. The eight-seed record check matters as much as the prompt. So do clean baselines and logs that let an agent distinguish a failed idea from broken code. Without those controls, a long autonomous run can generate an impressive volume of experiments while learning from contaminated results.
It also argues for keeping a person responsible for scope and interpretation. NanoGPT Speedrun Frontier has a scalar target and cheap feedback compared with most production research. Database tuning, compiler work, drug discovery, and large-model training expose more objectives and more ways for a local improvement to cause damage elsewhere. Prime Intellect's result shows that agents can navigate one constrained search space for days. It does not show that they can choose the right research question or recognize every omitted constraint.
The next useful evidence will come from the reruns and replications. Prime Intellect says it plans to add more seeds, models, and harnesses, and to explore partial internet access. Watch whether the ordering survives those controls, whether independent teams can reproduce the submitted recipes, and whether any agent produces a method that was absent from existing work. Until then, the strongest result is about experimental discipline: under a strict verifier, some agents can turn many noisy trials into a measurable gain, while originality remains unproven.