At 01:30 UTC on September 18, Astra for Law had drawn 449 Hacker News points and 467 comments, more comments than points. The number developers should keep beside that attention is 54.0%. OpenAI says its new legal system passed the overall correctness check on 54.0% of a 200-question legal research set at the highest reasoning effort. Plain GPT-6 Astra with web search passed 38.7%. That is a 15.3-point gain, yet the configured system still missed at least one required element on 46% of the questions.
The result explains both the interest and the unease around this launch. Domain search moved the score materially. It did not remove the need to inspect authorities, check citations, and decide whether the answer fits a client's facts. For teams building legal software, review is part of the workflow that Astra for Law requires.
A model wrapped in a legal stack
OpenAI has combined GPT-6 Astra with a legal search index, instructions for legal analysis and writing, privacy controls, and integrations for law firms. The company describes Astra for Law as a foundation for firms and legal technology vendors to build on, rather than a consumer legal adviser. Harvey and Legora are named as API customers that will be able to put it inside their own products.
Initial access is narrow. The OpenAI Help Center says selected U.S. law firms can request early access through a Trusted Access program. Eligible users must be lawyers or people working under their supervision. The product appears in ChatGPT and Codex as GPT-6 Astra Law. OpenAI says an API model named gpt-6-astra-law is coming soon. The announcement gives no date, public price, request schema, or rate limit for that API.
That packaging matters. OpenAI is selling a configured system whose behavior depends on retrieval and firm data, not merely a new checkpoint with legal branding. Its own comparison holds the base model constant and changes the tools around it. The strongest claimed improvement therefore belongs to the complete setup.
The 54% result needs its split
Vals AI's Legal Research Bench contains 413 U.S. legal questions written and peer-reviewed by practicing lawyers. Five are public samples, 200 form a private validation set available for license, and 208 are kept as a private test set. OpenAI evaluated Astra for Law on the 200-question validation split.
That distinction limits what the launch score can prove. Vals uses the 208-question test split for its public leaderboard specifically to reduce overfitting. A validation set is the part an organization can use while choosing prompts, tools, and other settings. OpenAI's launch materials do not report a result for Astra for Law on the held-out Vals test split, so its 54.0% cannot be compared directly with the scores on Vals' public board.
The grading rule is demanding. Each question has between one and 31 required rubric items, with an average of 9.35. A response gets the primary all-pass score only when it satisfies every item. Missing one requirement turns that question into a zero, even if the rest of the answer is useful. Vals says GPT-5.4 performs the rubric grading and agreed with a majority of three human expert reviewers 87.4% of the time in its validation, compared with 83.4% average agreement for one human reviewer against that majority.
This means the 46% failure share should not be read as 46% fabricated or worthless answers. It means those responses failed at least one part of a strict checklist. OpenAI did not publish a partial-credit score for its 200-question run, nor did it provide a practice-area breakdown. The top-line result says the full answer cleared the bar a little more than half the time.
Search accounts for much of the gain
The legal index covers U.S. case law, statutes, regulations, court rules, and administrative decisions across more than 230 million URLs, with new sources added daily. URLs are not the same thing as court decisions. One contributor, the nonprofit Free Law Project, says CourtListener holds more than nine million decisions from over 2,000 courts and covers more than 99.9% of published U.S. precedential case law. OpenAI says the new index complements licensed services such as Thomson Reuters rather than replacing them.
Against GPT-6 Astra using ordinary web search, Astra for Law found 24% more reference cases on case-law questions, according to OpenAI. On an audited set of target passages, it retrieved up to 54% more relevant passages from the correct opinions at the same reasoning effort. Those retrieval gains sit beside the 15.3-point rise in overall correctness. The comparison suggests that finding the right source and passage is a large part of the product's value.
Retrieval does not settle conflicts between sources. Vals reports that questions requiring models to reconcile jurisdictions, courts, or legal regimes reduced every tested model's all-pass score by between 6 and 17 points. The pooled drop was 11 points. A legal product therefore needs to preserve the court, jurisdiction, date, and quoted passage behind a claim. It should also make conflicting authority visible instead of compressing the disagreement into one confident paragraph. That is an engineering consequence of the benchmark's hardest failure pattern.
The API promise is still unfinished
OpenAI's launch reaches beyond research answers. It includes 26 partner-built plugins for tools used by firms and nine community plugins carrying 47 adaptable skills. The examples connect ChatGPT to systems such as iManage for matter files and DeepJudge for prior-deal comparisons. OpenAI's plugin documentation says workspace administrators can control access by role, review a plugin's permissions, and inspect usage through audit logs. It also says plugin data is not used to train models by default for Business, Enterprise, or Edu customers.
Each connection adds another permission boundary. A research answer may combine public case law, a client's private documents, and data returned by a partner system. Developers will need to retain source identity through that path so a reviewer can tell which assertion came from which store. An answer with citations can still be unsafe if a plugin retrieved a document the user was not meant to see. OpenAI says firms can define permitted sources and review processes, but the launch page does not publish a common permission model for all 26 legal plugins.
The privacy terms also vary by surface. OpenAI says eligible firms receive Zero Data Retention on its API, while use of ChatGPT Enterprise is excluded from human review by default. Those are separate statements. They do not establish that every ChatGPT, Codex, API, and plugin path has identical retention behavior. Buyers will need the exact data path and contract for the product they deploy.
OpenAI gives three examples of custom firm systems already built with its engineers. Sullivan & Cromwell made an agreement analyzer that compares a new deal with playbooks and selected precedents, then proposes redlines and draft advice for lawyers to review. Ropes & Gray built a diligence system that traces findings back to data-room sources. Cooley's GO Public carries deal changes across an IPO filing so lawyers can examine the effects. In each case, the useful unit is a reviewable work product with provenance, not an unexamined answer.
The debate is really about review
The Hacker News discussion is evidence of community interest, not an independent test of Astra for Law. Commenters argued over whether checking generated work saves time, how firms might bill for automated tasks, and whether junior lawyers lose training when software produces the first draft. Several anecdotes concerned other products and cannot establish how Astra performs.
The benchmark supplies firmer ground for one part of that argument. A system can save research time while still needing a lawyer to verify every authority and catch missing rubric items. Whether that is economical depends on the error pattern: five minutes checking clean citations is different from rebuilding a chain of reasoning after the model chose the wrong jurisdiction. OpenAI has published an aggregate pass rate, but not review time, cost per question, citation-error frequency, or the share of answers that failed because one item was missing.
The next useful numbers
The API identifier is known. The deployment evidence is not. Watch for a release date and price, followed by an external run on Vals' 208-question held-out test split. Per-practice scores, latency, citation accuracy, and reviewer time would show whether the legal index produces a dependable workflow or shifts labor from research to verification. Plugin audits should also state what each integration can read and write.
The 449-point burst shows how much attention legal AI can command. The 54.0% validation score sets a more useful baseline. Until OpenAI reports results on unseen questions and documents the API's controls, developers should treat gpt-6-astra-law as a supervised research component whose output must remain traceable to the source.