mrkeyoor.com_
Fri 04 Sept 03:34 UTC
AI6 min read

GPT-6 Astra Scores 99.9% in One Harness, 62.7% in Another

ARC Prize recorded a 37-point gap between Astra's best provider-adapter and standard-harness scores. The agent runtime made much of the difference.

By 01:30 UTC on September 4, OpenAI's GPT-6 Astra launch had drawn 1,305 points and 1,020 comments on Hacker News, less than seven hours after the submission appeared. The number explains the rush around the release. A more useful figure for developers came from ARC Prize's verified testing: Astra's best ARC-AGI-3 score was 62.7% with the standard harness and 99.9% with OpenAI's provider adapter. The model looked almost finished with the benchmark in one setup and well short of that mark in the other.

OpenAI introduced Astra as its new top model for computer use, coding, science and professional work, with an initial rollout to a limited set of organizations. Its launch page leads with a 99.9% ARC-AGI-3 result, alongside 98% on FrontierMath Tier 4 and 100% on ExploitBench. ARC Prize's split result changes how that first number should be read: the agent runtime and its handling of state contributed heavily to the score.

A 37-point harness gap

ARC-AGI-3 asks an agent to learn unfamiliar, game-like environments through interaction. The agent receives no instructions, rules or stated goal, so it must explore, infer the mechanics and carry what it learns into later actions. ARC Prize describes its standard harness as a provider-neutral interface in which the model chooses what to preserve in visible notes. The provider adapter can also retain OpenAI's opaque reasoning state between requests and compact long conversations.

Those setups produced very different ceilings. The verified results put Astra's best standard-harness run at 62.71% using max reasoning, at a reported cost of $26,098. Its best provider-adapter run reached 99.95% using high reasoning and cost $18,817. These are separate configurations rather than a controlled one-variable comparison, but they establish that the headline score does not describe Astra under every interface.

The adapter also changed efficiency. Across 167 game and reasoning-level pairs solved by both setups, ARC Prize reports that provider-adapter runs used 49% fewer tokens and took about 3.66 times less recorded elapsed time in aggregate. Astra used fewer actions than the human baseline on 96% of completed levels and 51.7% fewer actions per level on average. Those measurements connect context handling to both benchmark score and operating cost.

ARC Prize also watched the model build its own working machinery. In one maze-like environment, Astra created parsers, state models and separate solvers for navigation, combat and moving patrols. The organization says those tool-assisted runs measure the model together with its tools, under conditions unlike its controlled human study. It also says ARC-AGI-3 has deterministic, closed-ended mechanics and that saturation is not proof of AGI.

The API exposes more of the agent runtime

Astra's API changes fit the harness result. OpenAI's model guide adds asynchronous tool calls, mid-turn steering and reasoning-effort changes that can preserve the prompt cache. These are orchestration features. They decide what the model can do while a slow job runs, how a user correction enters an active task and whether earlier work survives a change in reasoning level.

For an asynchronous function, the tool-calling documentation tells developers to set async: true in the tool definition. The smallest useful part looks like this:

{
  "type": "function",
  "name": "get_weather",
  "async": true,
  "strict": true,
  "parameters": {
    "type": "object",
    "properties": { "city": { "type": "string" } },
    "required": ["city"],
    "additionalProperties": false
  }
}

The flag does less magic than its name may suggest. According to the async tool-calling documentation, the application still executes and tracks the job. Astra can continue reasoning or handle independent work, then the application sends the result in a later Responses request with the original call_id. A coding agent could start a slow test suite, inspect another file while it runs and incorporate the test result when it arrives. The application remains responsible for concurrency, errors and matching outputs to calls.

Mid-turn steering works over a WebSocket connection to the Responses API. A client can send a correction or new requirement against the active response, and the server creates a continuation after finishing the current output item and any hosted tool work already underway. Steering does not erase output already delivered, reverse an action or cancel a tool that has started. Developers therefore still need cancellation and compensation logic for operations with side effects.

OpenAI is also changing context handling inside Codex. The launch announcement says Astra can keep notes across context windows and search earlier windows for requirements or test results that were omitted from those notes. The feature is experimental and must be enabled in config.toml; OpenAI says it will become the default for Astra in the coming weeks. This is the product version of the lesson in ARC Prize's adapter test: preserving the right state can alter the result as much as another round of model tuning.

The token price is 2.5 times higher

Astra charges $10 per million input tokens, $1 per million cached input tokens and $50 per million output tokens. OpenAI's model comparison lists GPT-5.6 Sol at $4, $0.40 and $20 for the same three categories. Astra is therefore 2.5 times the per-token price across each one. A request consuming one million uncached input tokens and producing 100,000 output tokens would cost $15 on Astra and $6 on Sol, before tool fees or alternate processing rates.

The context specifications have not grown with that price. Both models list a 1.05 million-token window and a 128,000-token output limit. Astra's model page also says prompts above 272,000 input tokens are billed at twice the input and cache rates, plus 1.5 times the output rate, for the full request. Fast mode can cost twice the standard rate, and OpenAI says it is unavailable for Astra when an API project uses EU data residency.

OpenAI's model guide argues that Astra can cost less per completed task because it uses fewer output tokens on several evaluations. ARC Prize's provider-adapter result gives that claim some outside support for one unusual workload, where fewer tokens arrived with a higher score. It does not settle the economics for code review, browser automation or document work. Teams need to count retries, tool calls, elapsed time and human corrections on their own tasks; a 2.5 times token premium leaves little room for a vague quality gain.

The strongest gains are agent-shaped

OpenAI's published coding table is uneven. On Terminal-Bench 4.0, Astra scored 57.9% against 37.3% for GPT-5.6 Sol. DeepSWE v1.1 moved from 72.7% to 74.1%, while the Artificial Analysis Coding Agent Index moved from 65.1 to 67.0. The larger jump came on OpenAI's internal database-migration tasks, from 42.7% to 63.9%. The company notes that its research and API evaluations may differ from production ChatGPT because the prompts and tools are different.

OpenAI's computer-use table shows the same emphasis. The company reports 72.6% for Astra on an offline subset of OSWorld 2.0, against 65.7% for Sol, with simulated task time falling from roughly 75 minutes to 40. Its internal AutomationBench score rose from 18.1% to 41.4%. Several evaluations are internal, so production results need testing in each deployment.

Access will also limit immediate comparisons. OpenAI's model guide says the initial rollout covers enterprises in its Trusted Access Program, with API and Plus, Pro, Business and Enterprise access due over the following days. Enterprise workspace access is off by default at launch. The 1,020-comment Hacker News discussion therefore began before broad access, limiting the production evidence available on launch day.

The next evidence should come from matched public-API tests: the same repository, tools, prompt prefix and completion criteria on Astra and Sol, with token spend, wall time, failed calls and manual repair recorded. ARC Prize's two-harness result shows why the harness must be named beside the model. If Astra's async tools and retained state cut retries enough to cover the higher rate, the deployment case will be measurable. If the gain appears only in provider-tuned setups, that limitation will be measurable too.

We reviewed this

  1. learn — our honest review
  2. computer — our honest review
  3. OSWorld — our honest review

Sources

  1. GPT-6 Astra discussion on Hacker News
  2. GPT-6 Astra: A new generation of intelligence
  3. OpenAI's GPT-6 Astra on ARC-AGI-3
  4. GPT-6 Astra ARC-AGI results
  5. GPT-6 Astra model
  6. OpenAI model comparison
  7. Using GPT-6 Astra
  8. Async tool calling