Some developers won't join an experiment if it means doing half their work without AI. That's a problem for METR, the research nonprofit measuring whether coding tools save time. Its February study update described growing resistance that was skewing who participated and which tasks they submitted.
That finding gives a concrete complication to Andrej Karpathy's October 11 argument that people's understanding of AI capability is pulling apart. Access and experience can differ sharply. Read alongside METR's account, his argument raises a measurement question: how do you study gains among people unwilling to attempt the same work without AI?
The post drew 529 points and 746 comments in MrKeyoor's October 12 Hacker News snapshot. The discussion includes accounts of useful automation alongside objections about maintenance and unreviewed code. Those are community reactions, rather than independent verification of Karpathy's claims. For developers deciding how much work to delegate, the useful question is how to compare those experiences with measured results.
A funnel drawn with rough numbers
Karpathy estimates that roughly 6 billion people have barely encountered large language models, while another 1 billion to 2 billion use free chatbots casually. He puts professional users of advanced models in mathematics and coding at about 20 million, and people with internal access to frontier systems at about 5,000. He explicitly calls the figures "napkin math." His post supplies no population study behind them.
He argues that agents can finish projects that previously took professionals weeks or months, and attributes differing experiences to task ambition, uneven ability, and access. He also describes large internal agent collaborations, without a reproducible evaluation of those systems. These are Karpathy's assessments. The post doesn't establish how many people belong in each tier or how reliably the most ambitious work succeeds.
The comparison group starts disappearing
METR's July 2025 randomized study recruited 16 experienced open-source developers and assigned 246 real issues to conditions allowing or disallowing AI. Participants worked on repositories they knew well, primarily using Cursor Pro with Claude 3.5 or 3.7 Sonnet when AI was allowed. They took 19% longer with the tools, despite expecting a 24% speedup before the experiment.
Even afterward, participants believed AI had sped them up by 20%. That discrepancy makes the study relevant to the current argument: direct experience can still produce a mistaken estimate of time saved. METR cautioned that the result concerned early-2025 tools in a particular setting. It didn't establish that AI slowed most developers, or predict what later systems would do.
The follow-up began in August 2025. METR's February 2026 update called the new data unreliable. Some developers declined participation. In surveys, 30% to 50% reported withholding tasks they didn't want to do without AI. Lower participant pay was another possible source of selection.
If participants exclude work where they expect AI to help most, the experiment misses potential gains. METR thought developers were probably getting more benefit by early 2026, but said the data offered only weak evidence for its size. Multiple agents also complicated time reporting: developers could work on other tasks while agents ran.
Saving time doesn't settle the value question
METR took a different approach in a survey published in May. It surveyed 349 technical workers between February and April 2026. Across its different questions, respondents reported median estimates of 1.4 to 2 times as much value from their work with AI. Their median reported speed increase was threefold. These were respondents' estimates, rather than gains observed in a controlled experiment. The distinction matters when agents change which work gets done. METR gives the example of a researcher quickly building a data dashboard that contributes relatively little to the project. The time saved on that dashboard could be large, while the increase in useful research output remains smaller. Speed and value answer different questions even when both estimates are sincere.
The survey used a convenience sample and had low response rates among people contacted by email. METR warned about selection bias and the difficulty of answering hypothetical productivity questions. Its results document substantial perceived benefits among these respondents. They don't measure global adoption or independently validate the dramatic time savings described in Karpathy's post.
A two-hour task has a particular meaning
Benchmark evidence needs its own translation. In METR's original March 2025 time-horizon research, task length meant how long a human professional needed to finish the work. The researchers examined how agent success changed as tasks became longer, finding that the duration associated with 50% success had approximately doubled every seven months over the historical period studied.
That result supports a specific claim about progress on the evaluated tasks. It also depends on the success threshold: completing work half the time and completing it predictably are different operational propositions. A developer considering unattended work needs to know which threshold a reported result uses. The original research's extrapolations were conditional on the trend continuing, rather than measurements of future systems.
METR's time-horizon methodology page makes the definition explicit. A two-hour horizon describes task difficulty relative to human completion time. It doesn't mean the agent ran for two hours. The tasks are largely self-contained technical assignments with clear scoring criteria, and the human comparison often resembles someone working with little prior project context.
The methodology page also warns that measurements above 16 hours are unreliable with its existing task suite. It stopped being actively updated in September 2026. Those limits matter when comparing a published chart with claims about newer systems working for weeks. The chart offers a defined measurement, rather than a current census of everything agents can do.
Passing tests still leaves work for maintainers
A separate METR evaluation published in August 2025 shows why the definition of completion can change the result. Researchers tested an agent using Claude 3.7 Sonnet on 18 issues from two large open-source repositories. Its average success rate against maintainer-written tests was 38%. Yet none of the 15 submissions selected for manual review could be merged as submitted.
Reviewers found deficiencies including missing documentation and inadequate test coverage. They estimated that even submissions passing the reference tests needed an average of 26 minutes of repair. Those repair times were subjective estimates made by comparison with human solutions, rather than timed fixes by the original maintainers. The sample was small, the repositories had demanding standards, and the agent used a basic setup. The results can't grade today's agents. They do demonstrate a gap between two ways of judging the same output: functional test success and readiness for a maintainer to accept the change. For a team evaluating generated code, that distinction is a reason to record the work remaining after the agent stops.
Compare the system doing the work
The software around a model changes the task it can attempt. In its engineering guide to agents, Anthropic distinguishes fixed workflows from systems where the model chooses its next actions and tools. It describes agents checking environmental feedback, including execution results, as they proceed. A chatbot answer and a tool-using development session therefore expose different kinds of behavior.
The guide also says agent systems can exchange higher cost and latency for better task performance. In its coding discussion, Anthropic treats automated tests as useful checks while retaining human review for wider system requirements. The article was first published in December 2024 and now carries a notice that much of its tooling discussion has changed. Its architectural distinction remains useful when describing what was evaluated.
Taken together, these studies and Anthropic's engineering guidance suggest a practical comparison for developers: specify the model and tool setup, then count the effort needed to reach an accepted result. Include the review and corrections. A report that an agent wrote the patch answers less than a record showing what a maintainer had to do before merging it. That's our inference from these studies, rather than a new estimate of AI's productivity effect.
METR's experiment update outlines alternatives including fixed-task and developer-level experiments. Watch whether those designs retain enthusiastic users and capture tasks they'd otherwise exclude. A result that includes those developers, with human review counted, would give teams a firmer basis for deciding what to delegate.