mrkeyoor.comAsk

anthropics/defending-code-reference-harnesstested 08 Oct 2026commit d3bea6b

defending-code-reference-harness review

Treat this repository as material for building an internal security workflow, with a named maintenance owner.

  • Install 34 s
  • Build 5 s
411/426passed, run exited non-zero
  • 37 packages
Dev Toolsevaluationupdated 01 Nov 2026
+14stars / 7d

Use Anthropic's Defending Code Reference Harness if your security team wants to build and maintain its own Claude vulnerability workflow. Anthropic explicitly says the repository is not maintained and is not accepting contributions. It combines interactive Claude Code skills with an autonomous reference pipeline configured for C/C++ memory vulnerabilities using Docker and ASAN.

Verdict

Treat this repository as material for building an internal security workflow, with a named maintenance owner. Our sandbox established installation and command outcomes, but Docker-dependent tests stopped before they could validate patch behavior. Anthropic explicitly says the repository is not maintained and is not accepting contributions. Adopt its design only if your team can supply the execution environment, validate targets, and review findings and patches.

We ran it

Lab card: what happened when we ran defending-code-reference-harness
Lab card from our sandbox run
Screenshot of defending-code-reference-harness (claude.com/blog/using-llms-to-secure-source-code)
claude.com/blog/using-llms-to-secure-source-code
Install✓ · 34s37 packages · 40 MB
Build✓ · 5s
Tests✗ · 19s411 passed · 14 failed · 5 skipped · 1 errors of 426 (pytest)
Known vulns0(pip-audit)
Repo161 files~14,975 lines of source · 1.7 MB · 0 CI workflows · tests dir

Answers from our run

Does defending-code-reference-harness build from source?

Dependencies installed in 34 seconds (37 packages), and the build succeeded in 5 seconds. We cloned commit d3bea6b into a clean Debian container with 3 CPUs and no project-specific setup.

Do defending-code-reference-harness's tests pass?

No. The project's own test command finished in 19 seconds and reported 411 passed, 14 failed, 5 skipped, 1 collection error (pytest).

Does defending-code-reference-harness have known vulnerabilities in its dependencies?

pip-audit found none in the dependency tree at the time of our run.

Who should not use defending-code-reference-harness?

Teams seeking a maintained, supported scanner they can deploy without owning the infrastructure.

What are the alternatives to defending-code-reference-harness?

Semgrep, CodeQL, OSV-Scanner. Treat this repository as material for building an internal security workflow, with a named maintenance owner.

Setup2/5On the visible 1-5 rubric, substantial setup work remains: installation succeeded, but missing Docker blocked tests and autonomous execution also requires sandbox and model configuration.
Docs5/5On the visible 1-5 rubric, the supplied README comprehensively explains the workflow and points readers to sandbox, customization, and patching documentation; linked procedures were not execution-validated.
Community2/5On the visible 1-5 rubric, the explicit refusal of maintenance and contributions limits the support a team can expect; popularity is not used as evidence of support.
Maturity2/5On the visible 1-5 rubric, this reference requires substantial operator ownership and explicitly leaves autonomous triage and patching incompletely solved.

Discussed on

  1. hnAnthropic's open-source framework for AI-powered vulnerability discovery539 points

Who it’s for

Application-security engineers evaluating the separation of discovery, independent verification, triage, and patch review.
C/C++ maintainers who can provide Docker, gVisor, model access, and target-specific builds.
Teams prepared to maintain their own fork and validate its behavior against their threat model.

Who it’s NOT for

Teams seeking a maintained, supported scanner they can deploy without owning the infrastructure.
Operators without Docker and gVisor who need autonomous execution.
Teams expecting the supplied C/C++ detector to cover another language or vulnerability class without customization.
Organizations expecting generated findings or patches to remove the need for human security review.

Setup reality

Our run on October 8, 2026 used commit d3bea6b in our sandbox. Installation succeeded in 34 seconds, with 37 dependency packages occupying 40 MB. The build step returned zero after 5 seconds; the pytest command returned exit code 1 after 19 seconds.

The attached stored parser summaries report 411 passed, 14 failed, 5 skipped, 1 error, and a total field of 426. Those fields do not reconcile into an exhaustive unique test count, so they should not become a pass-rate denominator. The raw footer separately reports 14 failed, 411 passed, 5 skipped, and 1 error in 12.04 seconds.

The listed failures span tests/test_patch.py and tests/test_patch_grade.py; tests/test_patch_grade_e2e.py also errors. Each listed failure or error reports a missing docker executable. This record does not establish a completed autonomous scan, successful gVisor isolation, model access, vulnerability discovery, or patch verification.

What our sandbox established

Our run on October 8, 2026 used commit d3bea6b in our sandbox. The full recorded commit is d3bea6b5793b5f3d59a75ebe69a58efa88383145. Installation succeeded, the build step returned zero, and the test command failed. This is a correction from the existing stored run, not a new execution.

Our harness records each command's success flag, exit code, elapsed wall-clock time, timeout status, and available log tail. It also stores parser summaries of test output. In this record, installation took 34 seconds, the build step took 5 seconds, and testing took 19 seconds. The empty build log tail and zero exit code do not demonstrate that a target image, deployment, or autonomous workload worked.

The MrKeyoor testing methodology explains this kind of provenance and its limits. Its supplied October 31 snapshot is background documentation; it does not establish that the October 8 run used that later procedure unchanged.

Docker blocked the patch tests

The raw tail lists four failures in tests/test_patch.py and ten in tests/test_patch_grade.py, plus an error in tests/test_patch_grade_e2e.py. Each reports FileNotFoundError for the docker executable. The failure therefore exposes an unmet execution prerequisite in our sandbox. It does not demonstrate whether those checks would pass with Docker available.

The attached stored parser summaries contain 411 passed, 14 failed, 5 skipped, 1 error, and a total field of 426. These are stored parser summaries, not verified exhaustive counts of unique tests. Their fields do not reconcile, and the repeated summary lines in the tail must not be counted again. Skipped tests remain skipped.

The raw pytest footer reports 12.04 seconds, while the harness records 19 seconds for the test command. These describe different recorded timings; neither replaces the other.

Pip-audit recorded zero advisories. Scan coverage is unknown: the record has a packages field of zero and no scannedPackages value establishing what was checked. This is not evidence that all 37 installed dependency packages were scanned or that the repository is secure.

Choose between static review and execution verification

The pinned README separates two useful entry points. Interactive threat modeling, scanning, and triage read and write files. Patching static findings also reads and writes files. The README requires reviewing and approving tool use in Claude Code and warns that static candidates on non-canary targets can contain more false positives because nothing is built or run.

The autonomous path executes target code. It requires sandbox setup and refuses agent-spawning operations outside gVisor unless explicitly overridden. Its reference detector uses Docker builds and ASAN for C/C++ memory errors. Moving to another stack means defining the finding signal, the proof-of-concept artifact, and the target build and execution process.

The verification boundary is worth studying: the README describes a separate grader reproducing the crash in a fresh container, with only the proof of concept crossing from the finding agent. That is documented design, not a result we demonstrated. Our run did not establish successful crash reproduction or the effectiveness of that boundary.

Start by defining the target's trust boundaries and reviewing the interactive artifacts. Before attempting autonomous execution, provide Docker, complete the documented sandbox setup, configure Claude access, and validate one target. Keep responsibility for severity, disclosure, and patch acceptance with a human owner. Anthropic says autonomous triage and patching remain open issues and verified patches are not always upstreamable.

Which alternatives answer a different need?

Semgrep supports static checks in an IDE, as a pre-commit check, and in CI/CD workflows. Choose it when the immediate requirement is integrating code checks into developer workflows rather than operating this autonomous reference pipeline. Its README warns that Community Edition analyzes within a single function or file and can miss security findings. We did not compare detection results.

The github/codeql repository contains the standard CodeQL libraries and queries that power GitHub Advanced Security. Consider it for a query-based analysis workflow, while checking the separate CLI license before analyzing closed-source code. The repository itself is not the CLI or engine, and our evidence does not establish comparative coverage.

OSV-Scanner finds existing vulnerabilities affecting project dependencies. It is a complement to investigating previously unknown source flaws: dependency advisory matching answers a different question. Choose it when known vulnerable dependencies are the immediate concern.

How to read the ratings

These are editorial assessments, not benchmark results. The shared scale is 1 for a major obstacle, 2 for substantial operator work or limitations, 3 for an adequate documented baseline, 4 for strong support with limited gaps, and 5 for comprehensive support within the dimension assessed.

Setup ease is 2 because Python installation succeeded but Docker blocked tests and autonomous execution needs additional infrastructure. Documentation is 5 for the breadth of the supplied README's workflow, sandbox, customization, and patching explanations; that score does not certify every linked instruction. Community is 2 because the project explicitly declines maintenance and contributions, regardless of attention. Maturity is 2 because it is a reference implementation with acknowledged gaps in autonomous triage and patching.

Alternatives

ProjectWhat it isPick it when
SemgrepStatic analysis with IDE, pre-commit, and CI/CD integration.Choose it for code checks in developer workflows; account for Community Edition's documented function-or-file analysis boundary.
CodeQLStandard analysis libraries and queries used by GitHub Advanced Security.Choose a CodeQL workflow when you want query-based analysis; obtain the separately distributed CLI and check its licensing for closed-source code.
OSV-Scanner gh↗Dependency scanning against known vulnerability information.Use it for known vulnerable dependencies, alongside source investigation when both kinds of coverage matter.

What people are saying

  1. [github-trending] anthropics/defending-code-reference-harness

Sources

  1. Defending Code Reference Harness README
  2. Anthropic security workflow blog post
  3. Issue 7: missing prerequisite summary
  4. Pull request 29: target supply-chain hardening
  5. Repository metadata

More dev tools reviews