Giving coding agents 10 minutes or more barely changed the failure rate in Real-SWE. The benchmark reports that 73.4% of longer runs failed, compared with 71.4% of runs that finished sooner. Across 640 attempts on private production code, the best model-and-harness combination resolved 38.8% of its tasks, and six of the ten tasks had overall pass rates below 15%. The time result is observational, yet it points toward a problem developers know well: an agent can keep working without finding the business rule it misunderstood. Specific Labs published the benchmark in September 2026.
Real-SWE combines eight model-and-harness configurations, ten tasks, and eight independent runs per task. Specific Labs says it licensed each codebase from a company and drew the assignments from work handled by its engineers. The report names no participating company. It describes them instead as products such as a consumer fintech platform processing more than 100,000 bank statements and an events app with more than 200,000 users. One full instruction is visible; access to the sample is offered by request. The report presents the resulting 640 scored rollouts as pass@1 measurements.
That setup attacks a known weakness in public coding benchmarks. The original SWE-bench contains 2,294 instances from 12 public Python repositories, so its issues, patches, and surrounding code can appear in training material. When OpenAI introduced SWE-bench Verified, it said public GitHub scrapes were likely to create contamination. Real-SWE withholds both the proprietary systems and their solutions, making direct memorization less plausible.
The benchmark changes what the agent can remember
A private repository denies the agent a familiar trail of issue discussions and merged patches. It must infer local conventions from the checkout and inspect the services available in its sandbox. Specific's example billing task asks the agent to handle tax modes chosen by each business, customer exemptions, destination pricing, rejected addresses, invoice settlement, and VAT registrations. The environment includes sandbox and production tax services, an InfluxDB ledger, and a NestJS application. The published instruction reads like a condensed internal ticket whose rules are scattered across systems.
The prompts are not unusually long. Real-SWE reports a median instruction length of 1,742 characters, below the 2,056 shown for FrontierCode and the 1,975 shown for DeepSWE. The reference changes are wider: a median of 11 edited files for Real-SWE, compared with six for each of those two benchmarks. Specific calculates that file count from eight repository-backed sample tasks, a narrower set than the ten tasks on its leaderboard. Those scope notes appear beside the comparison, and they matter when reading the medians.
The work also crosses infrastructure and business tools. Depending on the task, the sandboxes expose PostgreSQL, MySQL, MongoDB, Redis, Kubernetes, an AWS emulator, Slack, email, and project-management services. No single task receives every tool; Specific says each environment includes only what its workflow needs. That inventory helps explain why a plausible code patch can still fail when it is wired to the wrong service or misses a side effect.
A leaderboard with no 40 percent result
Fable 5.1 running in Claude Code ranks first at 38.8%. GPT-6 Astra in Codex CLI follows at 33.8%, with Gemini 3.8 Flash in Gemini CLI at 31.2%. GLM 5.3 reaches 28.8%, Grok 4.6 and Muse Spark 1.3 each reach 23.8%, Kimi K3 scores 18.8%, and GPT-5.6 Sol records 16.2%. Real-SWE shows 95% confidence intervals around those rates, which are based on only 80 attempts per configuration. Close positions should be read as estimates rather than a settled order.
The leaderboard scores paired systems. Most model names appear with a different harness, while Claude Code is used for Fable and GLM and Codex CLI is used for the two OpenAI models. A higher row cannot tell us whether the model, its harness, or their interaction produced the gap. That is also why Real-SWE does not support a clean Claude Code versus Codex verdict. Specific explicitly says it evaluates model-and-harness combinations to reflect how engineers use them.
Difficulty varies sharply across the ten assignments. The multi-region sweep has a 67.2% resolution rate and the API-keys-and-environments task reaches 65.6%. At the other end, no run solves the analytics stream reducer, while the tax-jurisdiction task reaches 3.1% and the linearizable scan reaches 4.7%. Six tasks sit below 15%. A benchmark this small can move substantially when one task is added, removed, or reweighted.
Per-task results are less tidy than the overall ranking. GPT-6 Astra passes all eight multi-region trials but none of its eight attempts on tax jurisdiction, the linearizable scan, or the analytics reducer. Fable passes every API-keys-and-environments trial and every entitlement-overage trial, then goes zero for eight on the analytics reducer. The trial grid shows that success on one company-shaped workflow does not transfer reliably to another.
The failures look familiar to code reviewers
Specific classifies failed submissions as unverified assumptions, missed requirements, integration errors, regressions, or changes made in the wrong file. These are labels assigned from observed submissions rather than a measure of model intent. Among Fable's 49 failed runs, 18 miss a requirement, 17 contain an integration error, and 12 rest on an unchecked assumption. GPT-6 Astra fails 53 runs; 18 rely on an unverified assumption and another 18 contain integration errors. The report publishes counts and definitions for each class.
The distribution changes by model. Of Grok 4.6's 61 failures, 41 miss a requirement. For GPT-5.6 Sol, 29 of 67 failures come from an assumption it did not verify. Gemini 3.8 Flash records integration errors in 27 of its 55 failures. Real-SWE calculates these percentages within each model's failures, so they should not be compared as shares of all attempts.
Those labels describe routine review findings. A patch can compile and still omit an exemption path, write the correct logic into an unused script, or update a service without preserving its callers. The benchmark's low scores therefore say more about repository comprehension and verification than syntax generation. That interpretation follows from Specific's task definitions and failure taxonomy, not from evidence that the models reason about errors in a human way.
The duration split deserves similar restraint. Real-SWE reports 70 failures among 98 rollouts under 10 minutes and 398 among 542 longer runs. It does not say that run length was randomly assigned, so the figures cannot prove that more time is useless. Harder attempts may simply run longer. What the counts do establish is narrower: failure remained common among agents that operated for at least ten minutes. The report prints both totals.
Spending more did not buy reliability
Estimated mean cost per rollout ranges from $2.50 for Gemini 3.8 Flash to $6.96 for Fable 5.1. GPT-6 Astra is estimated at $4.67, while GPT-5.6 Sol costs $2.65. The highest-scoring configuration is also the most expensive in this set, but the remaining order does not track price: GLM 5.3 costs an estimated $5.12 and finishes behind the cheaper Gemini configuration. Specific reports these as estimated costs, which may differ from a team's bill under another pricing or caching setup.
Output volume is just as uneven. The report shows about 117,000 output tokens overall for GLM 5.3, 94,000 for Gemini 3.8 Flash, 64,000 for Fable, 24,000 for GPT-6 Astra, and 23,000 for GPT-5.6 Sol. Yet GLM finishes fourth and Astra second. The effort chart gives teams a reason to measure completed work alongside token use instead of treating a longer trace as deeper work.
Private code improves the test and limits its audit
Each agent ran in an isolated sandbox, and Specific says verifiers were inserted only at grading time. Tasks use Harbor format. Harbor defines itself as a framework for sandboxed agent evaluation, with task environments and verifiers separated from the agent run. Hiding the checks reduces the chance that an agent edits tests to manufacture a pass, while isolation gives each attempt a clean workspace.
The same privacy that reduces memorization also prevents a reader from reproducing the leaderboard from the article alone. Specific publishes aggregate results, per-task trial outcomes, one detailed instruction, and a methodology summary. The repositories, full task package, verifier code, model settings, and trajectories are not available from the report page. Access to its sample is request-based. Until outside evaluators can inspect more of that material under suitable confidentiality terms, the numbers remain a vendor-run measurement.
Task selection adds another limit. Specific says it screened for companies with substantial usage, strong engineering teams, and demanding workloads, but it does not provide a sampling frame or named participants. Ten handpicked assignments cannot represent all enterprise software work. The benchmark does provide sharper evidence than a polished demo for the exact tasks tested. Its table records all eight attempts for every model-task pair, including the failures.
What developers can take from 640 runs
The actionable signal is in the error classes. Teams evaluating an agent on their own repository can score whether it checked its assumptions, covered every stated behavior, connected the change to the running path, and preserved existing behavior. Those checks map directly to Real-SWE's published taxonomy. A single pass rate from public issues says less about deployment readiness than a private evaluation built around a team's services and acceptance tests.
Real-SWE's paired-system methodology also argues against choosing a coding agent from a model name alone. Harness behavior, tool access, hidden company rules, and task mix are part of the measured system. The next useful evidence would include more tasks, the same model tested across multiple harnesses, fuller configuration records, and an audit path for the private verifiers. Watch whether later Real-SWE runs preserve the current failure pattern when the task pool changes. A stable answer there would matter more than a small reshuffle at the top of its eight-row leaderboard.