MaxxBench changes the instructions or starting code of 89 DeepSWE coding tasks to test whether AI agents can adapt when familiar work comes with different requirements. Gavin Lo and Justin Lo published the research on October 20, 2026. For developers choosing a coding assistant, it asks a useful question: does a strong benchmark result survive a small change in the job?
The published leaderboard shows uneven results. Claude Opus 5.5 scores 76.4% on MaxxBench, compared with a listed DeepSWE score of 74.2%. Gemini 3.8 Flash scores 53.4%, against 73.8% on DeepSWE. These are the project's reported results, not measurements by MrKeyoor. A score gap alone doesn't establish that a model memorized benchmark answers.
A filename can change the answer
One Testem task asks agents to split browser test reports into separate files. The revised instructions require launcher names in filenames to become lowercase, alongside Windows-safe character replacement. They also require repeated calls to the reporter's finishing method to be safe and the internal Testem launcher to produce no file.
Those details make adaptation testable. An implementation can create separate reports and still fail the request by preserving capital letters. The task page exposes the instructions and individual run outcomes, letting readers inspect the requirement behind a pass or failure rather than judge an aggregate percentage alone.
The authors also revised reference solutions and tests, and repaired problems in the original grading material. Their research account therefore describes more than a prompt-only experiment. We read the published material but did not rerun the benchmark, so this article cannot independently confirm its scores or behavioral classifications.
Check the behavior behind the reported number
A separate task for the scc code counter asks for bounded-memory processing that spills file records to disk. The requirement applies throughout multi-format output, while several output formats must remain byte-for-byte identical to the unbounded mode. The program must also print a statistic describing peak retained records.
That creates two distinct things to verify: the output's correctness and whether memory retention obeys the limit. A compliant-looking statistic cannot answer the second question by itself. The task instructions give teams evaluating their own agents a concrete pattern to borrow: specify the resource constraint separately from the summary the generated program prints.
The authors say future releases will need fresh task mutations as existing ones become familiar. Watch whether performance holds on those revisions and whether outside evaluators reproduce the results. For a team choosing an agent today, changed requirements from its own repository would make a useful companion to a public leaderboard.