Six command batches in 43 minutes is what agent latency looks like when the world refuses to pause. In one Brood War Bench match, Grok 4.6 at xhigh effort produced 11,138 reasoning tokens, never fielded a combat unit, and effectively spent the game between actions. For developers building agents around tools, that failure is more instructive than the logo at the top of the table: a good plan that arrives after the state has changed is a bad action. The benchmark's match notes make that cost visible frame by frame.
The experiment put 19 model-and-effort configurations into a round robin, which yields 171 head-to-head matches. Codex Astra at xhigh effort finished 18-0. The report also drew 232 points and 101 comments on Hacker News by the brief's September 20 fetch, a sign that developers found the setup worth arguing over. That count measures community attention. The match claims below come from the published matrix, recordings, game-engine data, and harness logs.
One match per pairing, one clear caveat
Every configuration played every other configuration once, according to the benchmark methodology. That produces a complete 19 by 19 matrix, but only one result for each pairing. The report does not publish repeated trials or error bars. A race matchup, an early tactical choice, or one long reasoning pause can therefore decide a row in the table. The standings describe these 171 games. They are not a settled ordering of the underlying models.
Within that tournament, the separation at the top was large. Codex Astra xhigh won all 18 games with a reported 12.6 actions per minute and an average cost of $10.54 per game. Astra medium went 16-2, while Claude Fable placed third at 15-3. At the other end, Claude Haiku won none of its 16 decided matches, and Grok 4.6's three effort settings combined for three wins and 47 losses in the leaderboard's decided-game counts. Those figures come directly from the leaderboard and head-to-head matrix.
The author's own assessment supplies the most important limit on those scores: none of the systems played beyond beginner level. Even Astra and Fable failed to build complex armies, defend simple attacks, or execute concrete strategies reliably. He says a beginner using a photon-rush strategy would beat every entry, though the matrix does not include that human baseline. An 18-0 agent can dominate this field and still be poor at Brood War.
The game keeps moving while the model thinks
Grok's 43-minute match is the sharpest example because the failure can be counted. Its xhigh run spent 11,138 reasoning tokens and issued six command batches. In two other matches, Grok xhigh produced three Marines without reaching the opposing base, while the medium setting made two Zealots and also stayed home. The report characterizes these games as failures to maintain an observe-and-act loop, rather than failed strategic ideas.
The actions-per-minute column tells a similar story, with some needed restraint. Grok xhigh managed 2.8 APM and Grok medium 3.2. Claude Haiku registered 0.3 APM. Codex Astra xhigh reached 12.6 APM, while Astra low hit 25.7. More commands did not guarantee a better result, but extremely sparse commands left agents unable to respond to a game that kept advancing. The table records both the activity rate and outcomes.
The effort rows do not produce a simple upward curve. Astra improved from a 77.8% win rate at low effort to 88.9% at medium and 100% at xhigh. Codex 5.6 Sol moved the other way: medium scored 72.2%, low scored 66.7%, and xhigh scored 61.1%. With one match per pairing, those differences cannot establish that extra reasoning caused a win or loss. They do show why an agent's thinking budget has to be tested inside the timing constraints of its environment, rather than judged only by the quality of a final answer.
A developer can translate that observation into a control rule. Give the agent a deadline for its next valid action, let it interrupt analysis when the state changes, and measure how old its observation is when a tool call finally runs. Brood War exposes the need because enemy units do not wait. The recorded matches show the same basic loop that any live agent needs: observe, decide, act, then check what changed.
Coordination broke inside the agents
Codex often created separate subagents for the economy, army production, and army control. The benchmark author found that they communicated poorly. An army controller would send each newly produced unit straight into combat, unaware that another subagent was trying to assemble a larger force. The result was a familiar beginner error, one unit walking into a defended base at a time. Several match recordings document those staggered attacks.
The Codex field notes change what the leaderboard means. The measured system includes more than a base model. It also includes the harness, effort setting, delegation policy, and whatever shared state lets subagents coordinate. The author reports that Codex performed better when he directed attack timing and helped its subagents work together. Human guidance changed the orchestration even though the model name did not.
Claude Fable's games expose another side of the same problem. Fable climbed the technology tree far enough to build Mutalisks in one win. In another, it added a Robotics Facility, Citadel of Adun, Observatory, and Templar Archives before winning. It still went 15-3 and remained at beginner level because building the right structures did not always become a coherent army. The game notes separate visible progress from reliable execution.
Per-game cost needs context
The cost column puts a dollar figure beside each score. Astra xhigh averaged $10.54 per game, below Astra medium at $15.11 and Astra low at $21.07. That ordering does not mean higher effort was cheaper in general. The report gives observed cost per completed match and does not normalize it by game duration, command batch, or a fixed objective. Different match lengths can leave two settings with different bills before model quality enters the comparison.
The other Codex families are just as uneven. Sol xhigh cost $3.23 per game at a 61.1% win rate, medium cost $5.12 at 72.2%, and low cost $9.23 at 66.7%. Luna xhigh averaged only $0.16 but won 38.9% of its games, while Luna low cost $0.42 and won half. These published cost and result pairs are useful observations from this tournament. Choosing an agent for a real workload would still require repeated runs against the same task and deadline.
A benchmark for closed-loop work
Many agent evaluations score an answer or artifact after the run ends. Brood War Bench scores consequences while the system is still reasoning. The harness ran matches in parallel on Freestyle virtual machines and saved engine data plus both agents' logs for each game. That design lets a reader connect a long reasoning trace to the empty base, idle army, or late command that followed it. The report explains that architecture and publishes replays for individual failures.
Real-time pressure also makes stale observations easy to spot. Codex sometimes sent a Probe across the map early, and opposing agents spent dozens of seconds deciding how to respond while other work stopped. The tactic succeeded even though the attacker was weak. It exploited the opponent's decision delay. In the author's field notes, this disruption worked more consistently than Codex's attempts at sustained production and long-term army planning.
The published design limits what can travel beyond the game. The tournament compares systems against one another, not against a fixed suite of expert scenarios. It has no repeated matchups, and the human skill claim is an informed judgment rather than a measured control group. The full matrix and recordings make the individual outcomes inspectable, but they do not turn 171 matches into a general measure of intelligence.
What to watch in the next run
A stronger follow-up would replay each pairing several times under fixed race assignments and report the distribution of decision latency beside win rate. Testing the same model through different harnesses would help separate model ability from orchestration. The current benchmark page already exposes effort settings, cost, APM, and match logs, so those additions would build on data the system records rather than replace the experiment.
For now, the 18-0 score is less portable than the 43-minute failure. Track the interval between an agent's observation and its next valid command, then inspect what changed in between. Brood War leaves an idle base on the replay when that interval grows too long. A production agent may leave only a stale tool call in a log, where the same delay is much harder to see.