The 113 tasks cover 5 languages and repository-scale changes
DeepSWE tests coding agents on 113 original tasks across TypeScript, Go, Python, JavaScript, and Rust. Each task points to an upstream project and asks for an observable behavior change, such as an API addition or parser fix. This resembles sustained repository work more closely than completing one isolated function. The tradeoff is interpretive: an aggregate score compresses many codebases, dependency stacks, and failure modes into one number.
A task directory contains metadata, the agent's instruction, an environment recipe, held-out tests, grader configuration, and a reference solution. The reference patch is for human checking and is not applied during grading. Since v1.1, the agent and verifier run in separate environments. Pier collects the agent's commit and applies its patch to a pristine grading container, reducing the chance that an agent can alter hidden checks.
What happened when we ran it
commit 0b9fabb produced no install, build, or test result in our sandbox. On 2026-09-03, the harness found no supported ecosystem for this Python repository and no Dockerfile. The fresh Debian container had 3 CPUs, 8 GB of RAM, no secrets, and no privileged access. We have no timings, package totals, passing-test count, or dependency audit for the checkout.
The documented format places an environment Dockerfile inside each task, while the quick start delegates execution to Pier. That layout did not match our standard repository harness. Cloning DeepSWE alone does not produce a benchmark run. You still have to install the external runner, obtain the task images, configure a model, and execute a selected task.
Pier and a model-provider key are part of the minimum run
The README installs datacurve-pier with uv and requires a version newer than 0.3.0 for v1.1 grading. Its examples export either an Anthropic or OpenAI API key before invoking mini-swe-agent. Pier can also drive Claude Code, Codex, Gemini CLI, and OpenCode. A deterministic subset can be limited to 10 tasks, a sensible first pass before spending money and machine time on all 113.
Pier gives an agent selected network access while keeping the task environment offline. The model can reach its provider while the repository cannot browse for answers. The README says all public leaderboard scores used Pier, mini-swe-agent, and Modal. A local CLI agent or another sandbox provider can answer an internal question, but it does not reproduce the published setup.
Open verifier reports make task-level inspection necessary
Open issue 75 describes a v1.1 task whose grader expected an exact English display name that the instruction did not require. A candidate used a reasonable shorter name, yet reconciliation treated the expected test identifiers as missing. Issue 76 reports another task where an agent-created test file collided with hidden test names and caused a correct patch to fail. Both reports concern the mechanism that converts working code into a score.
An independent audit in issue 52 credits v1.1 with earlier fixes while flagging a remaining reference-solution failure, incomplete exclusion disclosure, and missing per-trial receipts. It says the model patch and core grader artifacts were unavailable for regrading the leaderboard decisions examined. Inspect failures by task and rerun some reference patches before using DeepSWE to choose a model. A percentage cannot tell you whether a miss came from the model, environment, or identifier matching.
The published task images have an open arm64 blocker
Issue 48 reproduces an arm64 failure with Pier 0.3.0. The reporter found that per-task images were amd64-only, after which Docker's QEMU fallback crashed during agent setup. The issue remained open when checked. An Apple Silicon laptop or Graviton runner is a poor starting point unless you have confirmed a newer compatible image. The README's quick start does not state an architecture requirement.
Issue 81 identifies another reproduction gap: a paper describes a 9,000-second rollout timeout while repository configuration uses 5,400 seconds. The reporter asks for the exact commit and override used for the paper. Timeout policy changes how much work an agent can finish. Pin the task commit, Pier version, image, model, reasoning setting, and timeout beside any score you plan to compare later.
Apache-2.0 covers Datacurve's work, not every upstream project
The repository includes an Apache-2.0 license and a provenance file for all 113 tasks. Datacurve's license covers its specifications, verifiers, and curation, while upstream code keeps its original license. The file names the license for every task and says all are permissive. Downstream redistribution still has to follow each upstream project's terms.
GitHub records the last push on August 26, 2026. New model requests were filed on September 2, and the repository had 72 combined open issues and pull requests: 66 issues and 6 pull requests. The GitHub releases API returned no latest release, though the documentation names v1.1. Active maintenance sits alongside open questions about scoring, architecture, pricing, and reproduction.
DeepSWE is evidence for evaluators, not a turnkey model selector
The separate verifier and detailed provenance give DeepSWE a credible design. Our 3-CPU, 8 GB sandbox result leaves its setup unverified, and the open grader reports argue against reading the leaderboard as a buying table. Use the corpus if your team can rerun tasks, fix the environment, and investigate disagreements. Choose SWE-bench or Terminal-Bench when their task style better matches the work you need to predict.