mrkeyoor.com_
Thu 01 Oct 19:39 UTC
Dev Toolsevaluationupdated 26 Aug 2026

defending-code-reference-harness review

Anthropic's Defending Code Reference Harness is an example security workflow built around Claude Code skills and autonomous agents. It covers threat modeling, static review, crash verification, deduplication, reporting, and patch checking, with a reference pipeline aimed at C and C++ memory vulnerabilities.

+14stars / 7d
Verdict

Our harness build passed in 8 seconds and 411 of 426 tests passed, while 14 failures and 1 setup error all stopped on a missing docker executable. Use this repository as design material for a staffed security program that can own sandboxing, target configuration, model access, and human review. Do not adopt it as an unattended scanner: Anthropic explicitly labels it unmaintained and a reference rather than a product.

We ran it

Lab card: what happened when we ran defending-code-reference-harnessScreenshot of defending-code-reference-harness (claude.com/blog/using-llms-to-secure-source-code)
Install✓ · 18s35 packages · 39 MB
Build✓ · 8s
Tests✗ · 20s411 passed · 14 failed · 5 skipped · 1 errors of 426 (pytest)
Known vulns0(pip-audit)
Repo161 files~14,975 lines of source · 1.7 MB · 0 CI workflows · tests dir

Answers from our run

Does defending-code-reference-harness build from source?

Dependencies installed in 18 seconds (35 packages), and the build succeeded in 8 seconds. We cloned commit d3bea6b into a clean Debian container with 3 CPUs and no project-specific setup.

Do defending-code-reference-harness's tests pass?

Not all of them: 411 of 426 passed and 14 failed when we ran the project's own test command (pytest), with 1 collection error. Some failures need services or credentials a bare container does not have.

Does defending-code-reference-harness have known vulnerabilities in its dependencies?

pip-audit found none in the dependency tree at the time of our run.

Who should not use defending-code-reference-harness?

Teams seeking a maintained scanner with product support: the README says this repository is not maintained and does not accept contributions.

What are the alternatives to defending-code-reference-harness?

Semgrep, CodeQL, OSV-Scanner. Our harness build passed in 8 seconds and 411 of 426 tests passed, while 14 failures and 1 setup error all stopped on a missing docker executable.

Setup2/5Python setup is quick; Docker, gVisor, targets, and Claude remain
Docs5/5Pipeline, sandbox, prompting, customization, and patching are explained
Community2/57,365 stars, but the README declines maintenance and contributions
Maturity2/5Reference workflow with explicit gaps in triage and patch operations

Discussed on

  1. hnAnthropic's open-source framework for AI-powered vulnerability discovery539 points

Who it’s for

Application-security teams designing an internal Claude-based scanning workflow.
C and C++ maintainers who can run ASAN targets inside Docker and gVisor.
Security engineers who want reusable prompts, artifact formats, sandbox controls, and staged verification.
Claude Code users who prefer guided skills for threat models, scanning, triage, patching, and customization.

Who it’s NOT for

Teams seeking a maintained scanner with product support: the README says this repository is not maintained and does not accept contributions.
Users who cannot provide Docker and gVisor for autonomous runs: the pipeline refuses to spawn agents outside its sandbox unless explicitly overridden.
Java, Rust, or web-application teams expecting an immediate fit: the supplied dynamic harness is configured for C and C++ memory bugs with ASAN and requires customization for other languages or signals.
Organizations expecting static findings to be confirmed automatically: the README warns that the interactive static scan can produce more false positives on non-canary targets.
Teams wanting autonomous triage and patches to replace security review: the project says those remain open problems and severity depends on the deployment environment.

Setup reality

Our Python install succeeded in 18 seconds, adding 35 packages and using 39 MB. The build passed in 8 seconds. Tests failed after 20 seconds: 411 passed, 14 failed, 5 skipped, and 1 had a setup error out of 426.

Every listed failure and the end-to-end error raised FileNotFoundError for docker. A full autonomous run also needs one-time gVisor sandbox setup and Claude access through an API key, OAuth token, Bedrock, Vertex, or Azure configuration.

The 1.7 MB checkout held 161 files and about 14,975 source lines. It had no CI workflows, no root Dockerfile, and a tests directory. Pip-audit found 0 known vulnerabilities. Target images, ASAN builds, egress rules, and codebase-specific configuration remain operator work.

Six Claude Code skills cover the interactive security loop

The repository starts with guided work that a security engineer can inspect as it happens. Its Claude Code skills create a threat model, scan source, triage findings, draft patches, and customize the pipeline. The static path reads and writes files without building the target. It produces named Markdown and JSON artifacts, which makes the handoff between stages visible and gives a team something concrete to review or integrate into an existing case system.

That path has a deliberate limit. The README says static findings on ordinary targets can contain more false positives because no candidate is built or executed. A canary fixture exists to demonstrate confirmation and deduplication, but a planted vulnerable file can also be dismissed as test code. The useful lesson is procedural: define exposure first, preserve evidence, and keep triage separate from discovery. The skills help enforce that order; they do not turn model output into a confirmed vulnerability.

The autonomous path uses 7 stages and a gVisor boundary

The full pipeline builds a C or C++ target with ASAN, maps attack surfaces, runs parallel find agents, verifies each crash in a clean container, deduplicates it, writes an exploitability report, and grades a proposed patch. A crash must reproduce 3 out of 3 times before it advances. Only the proof-of-concept input crosses from the finding agent to the separate verifier, reducing the chance that hidden workspace changes manufacture a result.

Autonomous agents execute target code, so the launcher requires a gVisor sandbox unless the operator overrides the check. Setup builds agent images and restricts outbound traffic to the Claude API. Target directories are still trusted executable configuration: they define images, build commands, test commands, and harness behavior. A safe deployment must review those files, limit mounted paths, protect host credentials, and keep the egress allowlist narrow.

What happened when we ran it

Our sandbox installed 35 Python packages in 18 seconds and used 39 MB. The build then passed in 8 seconds. Pytest ran for 20 seconds and exited with code 1: 411 tests passed, 14 failed, 5 were skipped, and 1 hit a collection or setup error out of 426. Pip-audit found 0 known vulnerabilities in the installed Python dependencies.

The failure tail was consistent. Fourteen test_patch_grade cases raised FileNotFoundError: [Errno 2] No such file or directory: 'docker', and tests/test_patch_grade_e2e.py failed during setup for the same reason. Our unprivileged container did not provide Docker. The log does not show a Python assertion or grading mismatch behind those cases, but absence of Docker is still a real setup failure for a project whose dynamic verification depends on containers.

The reference is small, while each target carries the hard work

The checkout was 1.7 MB with 161 files and roughly 14,975 source lines. It had a tests directory, no root Dockerfile, and 0 CI workflow files. Its installation added only 35 packages. Those figures make the harness code readable enough to study, but operational cost moves into target images, sanitizers, repeated model calls, result storage, and the engineering needed to define a trustworthy crash signal.

The supplied reference uses ASAN for C and C++ memory errors. Porting it means deciding what counts as a finding, what proof of concept the verifier receives, and how the target builds and runs. An HTTP request sequence might replace a crashing file; an exception or canary write might replace an ASAN signature. Those choices determine whether the system verifies the vulnerability you care about or rewards an agent for producing an unrelated failure.

Claude access and source handling need a written policy

A run needs access to Claude through an Anthropic API key, a Claude Code OAuth token, or supported Bedrock, Vertex, or Azure configuration. The sandbox's egress policy controls where agent traffic can go, but the model still needs enough code and runtime evidence to perform its task. Security teams should decide which repositories may be processed by each provider route, how transcripts and findings are retained, and who can approve patches.

The /patch skill is safer on static artifacts because it only reads and writes files. When given pipeline results, it executes target code to grade the fix and therefore returns to the gVisor boundary. Even after a patch passes its ladder, the README warns that verified patches are not always suitable for upstreaming. Ownership, regression risk, disclosure, and business severity remain human decisions.

The repository is explicitly unmaintained despite recent GitHub activity

GitHub reported 7,365 stars, 23 combined open issues and pull requests, and a last push on August 6, 2026. No latest release endpoint existed. Several open pull requests from July propose fixes to sandbox parsing, target supply-chain controls, argument construction, and artifact validation. That activity does not override the first-page policy: Anthropic says the repository is not maintained and will not accept contributions.

This makes the harness useful architecture and risky infrastructure. Copy its separation of threat modeling, finding, independent verification, deduplication, and patch grading. Keep the gVisor refusal and restricted egress. Then own a fork, dependency updates, tests, model changes, and target definitions as an internal security system. A team unwilling to take that maintenance burden should use a supported scanner or Anthropic's managed product instead.

Alternatives

ProjectWhat it isPick it when
SemgrepA rule-based static analyzer for finding code patterns across many languages.pick this instead when deterministic rules, broad language coverage, and CI-friendly static checks matter more than agent-led investigation.
CodeQLA semantic code-analysis engine driven by queries over a program database.pick this instead when repeatable data-flow queries and established language models are the main requirement.
OSV-Scanner gh↗A scanner for known vulnerable dependencies and supported source or image inventories.pick this instead when the immediate job is matching dependencies to published advisories rather than discovering new code flaws.

What people are saying

  1. [github-trending] anthropics/defending-code-reference-harness

Sources

  1. Defending Code Reference Harness README
  2. Anthropic security workflow blog post
  3. Issue 7: missing prerequisite summary
  4. Pull request 29: target supply-chain hardening
  5. Repository metadata

More dev tools reviews

nyaterm · yoinks · tilelang · vintage-latex · NavierStokesAndEuler · UMR · the whole board →