A 27-billion-parameter model beat Claude Opus 4.8 and GPT-5.5 on 60% of a set of held-out research-replication tasks, but the smaller model did not write the experimental code alone. It managed GPT-5.5 Codex as a tool, then had its work graded by another Codex-based system. That division of labor is the useful part of Inherent's Faraday result: a modest model trained to make research decisions can improve what a much larger coding agent produces. It also makes the benchmark's judge, rather than the headline model comparison, the part that deserves the closest inspection.
The London lab released Faraday alongside Replica, a suite of 310 figure-replication tasks drawn from 100 machine-learning and AI-for-science papers published between 1990 and 2026. TechCrunch reported the launch on August 22, shortly after Inherent emerged from stealth with a $50 million seed round. The company's larger ambition is an agent that contributes to scientific discovery. The evidence published so far supports a narrower claim: reinforcement learning taught Faraday to organize constrained attempts to reproduce a paper's result.
Replication without the answer image
Each Replica task removes a results figure from a paper while leaving its caption and the rest of the paper available. The agent receives a container with scientific libraries, internet access, a slice equal to one-seventh of an H200 GPU, and a fixed deadline. It must infer what experiment produced the missing figure, implement a smaller version when the original is too expensive, and generate its own result. The 47-page research paper says early training runs lasted 30 minutes and later ones lasted an hour.
This setup tests more than whether an agent can turn a written method into code. Papers omit failed experiments, tuning decisions, implementation details, and compromises made under a compute budget. An agent can also produce a plausible-looking chart through a bad shortcut. Replica's grading rubric therefore considers whether the implementation preserves the mechanism behind the paper's claim, whether a scaled-down experiment still represents the stated scope, and whether the agent avoids hard-coded outputs.
The dataset split matters. Faraday trained on 242 tasks from machine-learning papers, while 68 tasks from AI-for-science papers were held out. Papers contributed between one and 13 tasks, with a median of two. The task builder used Gemini 2.5 Pro to locate and remove figures, after which people filtered failures such as incorrect figure localization or inadequate redaction. The evaluation is out of distribution by research topic, although it remains within the same figure-replication format.
The smaller model is a manager, not the coder
Faraday starts from Qwen3.6-27B. The outer model reads the paper, plans the work, decides what to delegate, and can call a coding agent repeatedly or in parallel. Most of its training used GPT-5.4 mini behind the Codex command-line tool; the final stages and evaluation used GPT-5.5. A wrapper preserved or reset coding sessions at Faraday's direction and returned command transcripts with timings.
That architecture changes how to read the size comparison. Faraday is smaller than the frontier systems in the benchmark, but it has access to one of them. The reported gain comes from adding a trained research controller above the coding model, not from replacing frontier compute with a standalone 27B model. In the paper's ablation, a version trained without the coding-agent tool performed worse and its training collapsed after roughly 300 steps, even though it had twice Faraday's time allowance during the comparable period.
The authors also tested whether the controller was tied to the coder it saw during training. A checkpoint trained only with GPT-5.4 mini scored better on held-out tasks when GPT-5.5 was swapped in at evaluation. That suggests the learned behavior includes tool direction that can survive an upgrade beneath it. For teams building research agents, this may be more practical than training a single model to contain experimental planning, software work, and domain knowledge all at once.
What the benchmark says, and what it does not
Every evaluated system received the same materials and a 60-minute, single-GPU budget. Claude Opus 4.8 ran in Claude Code and GPT-5.5 ran in Codex, both at extra-high reasoning effort. Faraday used GPT-5.5 as its coding tool at the same effort setting. The researchers ran eight attempts per task and averaged their scores.
On the training distribution, Faraday outscored both Claude and Codex on 73% of tasks. On the 68 held-out AI-for-science tasks, it did so on 60%. Its average test score was 6% above Claude and 8% above Codex, according to the project's rubric judge. Those numbers support a claim about this task format under this compute limit. They do not establish that Faraday can independently reproduce whole papers, arbitrate disputed findings, or generate reliable discoveries across science.
The team ran a second, much smaller test closer to full-scale work. It selected eight unseen tasks judged to require no more than eight hours and eight B300 GPUs, then gave Faraday and Claude the estimated resources. Faraday scored higher on five of the eight and had the higher average. The paper explicitly says the rubric was not validated with human ratings at that scale, and each task received only one run.
Another experiment asked the system to reproduce results from 20 invented papers. The judge preferred Faraday over Codex on 19 tasks. The authors call this a weak form of innovation, then warn that their judge was never validated for imagined research. That warning should travel with the 19-of-20 figure. A scorer designed around faithful replication may behave differently when there is no real paper result to anchor the work.
The judge is part of the experiment
Replica cannot use a simple unit test as its reward. Claude Opus 4.7 generated a task-specific rubric, and multiple GPT-5.5 Codex judge samples examined the redacted paper, original figure, generated plot, code, Git history, and full agent trace. Their scores were averaged. The judge also assigned credit to individual turns, giving reinforcement learning a way to reward an early experimental choice more than the routine commands that followed it.
The authors tested this judge against rankings from research experts. Two independent samples from the rubric judge had a Kendall rank correlation of 0.66. Two human rankings correlated at 0.30. The rubric judge's correlation with humans was 0.19, compared with 0.15 for a generic baseline judge. It was more consistent and somewhat closer to human ordering than the baseline, but 0.19 is still weak agreement. The paper acknowledges tasks where the automated and human rankings diverged.
This does not make the results empty. A low-noise reward can be useful for training even when it is an imperfect stand-in for expert judgment, and the researchers checked whether humans preferred Faraday in cases where the judge gave it an advantage. Still, Faraday learned from a reward produced by the same family of coding model later used to score its headline performance. Independent evaluation with a different judge, more human raters, and replications run by outside labs would show how much of the gain transfers beyond the benchmark's own definition of good work.
The training recipe also reveals how much engineering sits behind the apparent simplicity. The final lineage ran for 659 reinforcement-learning steps. It moved from one judge sample to three, introduced turn-level credit assignment late in training, and corrected captions in 17% of tasks during the last stage. Removing turn-level credit assignment in an ablation caused reward to collapse after about 50 steps. Faraday may use a simple runtime harness, but producing the model required a carefully adjusted training system.
Where a research controller could help
Inherent's strongest evidence concerns choices made under constraint. The paper reports that Faraday was more likely than the baselines to implement the causal mechanism, choose a faithful reduction in scale, and avoid shortcuts that made a result look better. Those are useful behaviors in computational research, where an apparently successful run can rest on leakage, a mismatched baseline, or a plot that conceals a broken comparison.
A controller that delegates implementation while retaining responsibility for experimental design could also make model upgrades less disruptive. The coding layer can change as stronger tools arrive, while the outer model keeps the task history and research strategy. Faraday's GPT-5.4-mini-to-GPT-5.5 swap offers early evidence for that arrangement, though one benchmark is too narrow to establish it as a general design rule.
The system is also confined to computational tasks selected by its creators as unlikely to cause harm. It had limited time and compute, and the paper cautions that a failed agent replication says nothing by itself about whether the original study is wrong. Some experiments fail because an agent makes a poor choice, misses an unstated detail, or cannot fit a faithful test inside the available resources. Human review remains necessary before any failure becomes a scientific claim.
The next evidence to watch is external: whether other groups can run Replica, reproduce the rankings with different judges, and test Faraday on full-paper work rather than missing figures. Inherent will also need to show that its controller reports genuine non-replication instead of optimizing for a rubric-friendly approximation. Until then, the result is best read as evidence for a research-manager layer above coding agents, not as proof that a 27B model has become an autonomous scientist.