Armature's experiment began with 16,893 coding-agent sessions, yet its first published analysis covers 5,292 of them, or 31.3%. Within that retained set, Claude Code, Codex and Cursor chose the same tool in only 42% of comparable cells. For a developer delegating the choice of a database, payment service or deployment host, the agent can influence the dependency before anyone reviews a benchmark or contract. The study's own results suggest that changing the agent may change the answer even when the task remains comparable.
The finding is more useful than a simple vendor leaderboard. It exposes a new source of variation in software architecture: each agent searches differently, reads a repository through its own model, then turns that interpretation into code. Armature also has a commercial interest in this question. Its research disclosure says the company sells growth services to developer-tool vendors and studies how products get picked by agents. That makes the public traces especially important for checking the claims.
How the experiment worked
Armature assembled 75 repositories in 10 programming languages. The projects use fictional company names, fake histories and fake API keys, while their lockfiles were checked against real package registries. The team then produced 1,163 prompt variations across four simulated personas, ranging from a user who describes a symptom to an enterprise engineer who specifies procurement or compliance constraints. According to the methodology, each run asked an agent to inspect a codebase, choose a service and implement it.
Each session ran in an ephemeral sandbox, rotated among E2B, Blaxel and Daytona. A Gemini 3.7 Flash instance played the user after the first response, accepting the top recommendation or asking the agent to choose and proceed. Armature says this step reduced a bias it saw when agents were told to implement immediately: without a chance to request approval, they built more features in house. A second Gemini 3.7 Flash instance judged the completed session and recorded the chosen product, other products mentioned and the code change, as the study explains.
The two Gemini roles belong inside any interpretation of the results. One shaped the conversation and another classified the outcome. Their decisions were part of the measurement chain, even though the coding work came from Claude Code, Codex or Cursor. The published methodology gives examples of invalid sessions, including repositories that had already settled on a provider and runs that never made a choice.
The retained sample was close to evenly divided by harness. The machine-readable data, generated on September 2, contains 1,764 Claude Code runs, 1,782 Codex runs and 1,746 Cursor runs. It also identifies the underlying models: most Claude Code sessions used Claude Opus 5, most Codex sessions used GPT-5.6 Sol, and every listed Cursor session used Grok 4.6. The reported behavior therefore describes these model and harness combinations at a particular time. A future update could move the numbers without any change to a developer's prompt.
Search behavior split the agents
Codex searched the web in 94% of sessions, according to Armature's analysis. Nine out of ten Codex queries used operators such as site: to narrow the search to a trusted domain or a particular product. Cursor based its decision on web material in about two-thirds of sessions. Claude Code searched in roughly 30%, relying more often on model priors, though its search rate rose to about 80% for newer categories such as agent sandboxes.
Claude Code opened about three times as many pages as Codex on the occasions when it did browse, the researchers reported. Those patterns offer one explanation for the 42% agreement rate. An agent that starts with remembered product associations is answering from a different evidence set than one that runs a domain-restricted query. The experiment cannot tell us which search style produces the best production decision because it scored what was selected and implemented, not how that service performed months later.
The agents also differed in their willingness to avoid an outside service. Claude Code built a solution in house in 19% of the published cases, compared with 10% for Codex and Cursor, according to the aggregate findings. That gap matters when a task can be solved by a package, a hosted API or a small local implementation. Tool choice begins one step earlier than the brand comparison: the agent first decides whether a vendor belongs in the solution at all.
The repository changed the recommendation
The same email-service request produced a different leader in four language contexts. Resend won 55 of 89 TypeScript runs, SendGrid won 22 of 24 Python runs, Postmark took 20 of 24 Go runs, and Azure Communication Services took 22 of 23 Java runs. The study's language breakdown does not isolate language as the sole cause, since frameworks and existing project structure travel with it. It does show that a general question such as "which email provider is best?" leaves out information that materially changed what the agents installed.
Deployment choices showed the same pull from surrounding code. Vercel won every experiment involving Next.js in the reported set, while it was never recommended for a Python repository, where Render led. These repository-specific results make a universal leaderboard hard to defend. Compatibility, existing conventions and the friction visible in a codebase all enter the answer before a product's broad reputation does.
That context sensitivity can be useful. An agent should notice an existing cloud account, runtime or database client. It can also reinforce defaults simply because documentation, examples and SDKs are easier to find for one stack. Armature varied codebases and prompt personas, but its research post does not publish population weights or confidence intervals beside the headline comparisons. The install shares should be read as observations from this panel, not estimates of global developer market share.
A mention did not predict an install
Several familiar names appeared often in deliberation and rarely survived to implementation. PayPal was mentioned 139 times and was never picked; Stripe won 124 of those sessions. Adyen appeared 175 times and won three. LangChain was cited 194 times but selected four times, while Netlify converted six of 152 mentions. Supabase led database mentions at 242, yet Neon won 66% of database runs overall, according to the published comparisons.
Armature attributes some losses to how product pages present tradeoffs. Its agents repeatedly reacted to Mailgun's one-day retention wording and treated Supabase's bundled auth, storage and realtime features as overhead when the request asked only for a database. Across 5,292 sessions, 388 mentioned management overhead and 195 mentioned cost. These are the research team's interpretations. The design does not prove that a copy change would reverse a choice. The traces do show what the agents wrote down while deciding.
For tool makers, this shifts part of developer relations toward material that machines can retrieve and compare. Clear compatibility notes, current pricing constraints and examples for a specific runtime may enter an agent's working evidence. The search data also warns against optimizing for citations alone: PayPal and LangChain frequently reached the candidate list without becoming the installed dependency.
What developers can take from it
A tool recommendation from one coding agent should be treated as a proposal when the dependency carries ongoing cost, security exposure or operational work. The 42% cross-agent agreement result gives teams a practical check: run the same repository constraints through a second model, then inspect why the answers differ. Armature's public leaderboard lets readers replay the prompt, searches, commands, code diff and judge verdict behind each included run, so the disagreement can be examined below the brand totals.
Prompts should also state the constraints that a human review would apply. Data residency, expected volume, an existing cloud contract or a ban on another stateful service can change the choice, as the persona and repository experiments demonstrate. Requiring primary documentation and a short decision record would make the recommendation easier to audit, especially when the agent otherwise relies on its stored priors.
The largest open question is the missing 11,601 sessions, about 68.7% of the original run count. Armature says those sessions remain under consideration for a later wave. The current article does not break the exclusions into numerical categories. The next useful release would publish that accounting, then rerun matched cells as the underlying models change. Until that happens, the study is strongest as evidence that agent choice is variable and inspectable. It is weaker as a final ranking of the services developers should buy.