Review the work product and the grading contract
Harvey Legal Agent Benchmark (LAB) gives an agent documents and instructions, then grades the files it produces. It is a candidate for teams evaluating document-based legal assignments who can also inspect the rubric behind each score.
The README describes two components: a task dataset and an execution harness. The architecture describes synthetic matter files, saved transcripts, metrics and deliverables, followed by model judging and static reports. These are documented capabilities; our recorded software-test run did not measure the quality of a legal deliverable.
What our recorded run establishes
Our recorded run took place on October 4, 2026, at commit 985955a35c54f1119f7f51f8a63a7e53da4f6bc4. Its configuration recorded 3 CPUs, 8 GB of memory and image ghcr.io/astral-sh/uv:python3.12-bookworm. The install tail separately identifies Python 3.12.12.
The install step exited with code 0 after 15.373 seconds; the dependency summary recorded 91 packages and 404 MB. The build-check step exited with code 0 after 1.448 seconds. Its empty recorded tail does not establish which distributable artifacts were produced or whether an installed binary was checked.
The test step exited with code 0 after 39.262 seconds. The pytest tail reported 12,870 passed, 64 skipped and 28 subtests passed in 34.06 seconds. The parser summary recorded zero failures and zero errors. Skipped tests are separate from failures. The supplied record warns that these summaries do not establish exhaustive unique test coverage; no new execution or complete-log reconciliation was performed.
The checkout summary recorded 63,086 files, approximately 16,333 source lines, 3,223.9 MB and four CI workflows. The pip-audit summary recorded zero advisories, but audit coverage is unknown. It does not establish how many packages were inspected or prove dependency safety.
This record supplies no measured provider-backed agent assignment, judge workload, legal accuracy result or cost benchmark. Successful dependency installation, a build-check exit and pytest execution should each retain that narrower meaning.
Setup extends beyond Python dependencies
The README documents the harness and grader as the lab-core Python package, with import name lab_core, and provides a v1.2.0 wheel. It says to point LAB_ROOT at a task checkout to run its Python module commands outside the repository. Its research citation still names v1.0; that citation is distinct from the documented wheel version.
For DOCX grading, the README requires Pandoc 3.5 or later on PATH. The architecture lists document parsers appropriate to different file types and describes six workspace tools: bash, read, write, edit, glob and grep. It also documents a separate finish completion signal. The completion check verifies the agent's listed output paths, rather than validating every deliverable required by task.json.
According to the architecture, the six workspace tools route through a per-task Podman sandbox with networking disabled, capabilities dropped and source documents mounted read-only. This describes the intended isolation design; our stored run is not a security assessment or proof that a task sandbox was exercised successfully.
Understand the dual-judge score
The evaluation documentation names Claude Sonnet 4.6 and GPT-5.5 as the default independent judges. Each judge assesses each criterion against its match_criteria text using only the deliverables assigned to that criterion. There is no separate golden reference output.
For each judge, a task receives 1.0 only when every criterion passes; otherwise it receives 0.0. The standard dual result averages the two judges' task scores, so a single task's dual all-pass rate can be 0.0, 0.5 or 1.0. The aggregate all_pass flag requires both judges to all-pass. Criterion pass rates remain useful diagnostics, but they answer a different question from all-pass rate.
This distinction matters when reading a dashboard: a dual value of 0.5 can reflect disagreement between judges about whether the task passed completely. It does not mean that half the legal work was correct. Preserve both judges' reasoning and check the criteria responsible for disagreements.
Separate task reports from findings in our run
The supplied page for issue #152 is open. Its author reports that five of one task's 49 criteria refer to a $31 million escrow absent from the nine supplied documents and instructions, with three criteria requiring that figure directly. This is a contributor's task-specific report, not a defect we reproduced in our sandbox.
The supplied page for issue #146 is also open. Its report concerns seven firm-knowledge task contracts at commit 7be41d57fd5a6e97b5f246a029e810f83d09cd96, rather than our measured commit. The author describes fixed counts that conflict with permitted matter lists and explicitly does not claim a measured solver-plus-judge score delta. The extract does not establish whether those same contracts remain unchanged at 985955a.
These reports give teams concrete reasons to check whether instructions, documents and criteria agree. Their open status does not establish that every reported defect affects every current task.
The converter report has a later resolution
Issue #145 reported a historical failure mode in which an unreadable deliverable became error text that a judge then graded as the agent's work. Its author reported 7 of 50 criteria passing without Pandoc and 34 of 50 after installing it and rescoring the same artifacts. Those figures belong to the issue author's experiment, not ours.
The supplied issue page records closure as completed through #178 on October 19, 2026, with a referenced change titled "[grading] Stop grading unreadable deliverables, record them as errors (...". That closure postdates our October 4 run. It supports describing the issue as resolved in the supplied issue history, but does not establish that our measured commit included the change or that we tested it. Check converter availability and inspect extraction errors before interpreting a legal score.
How to read our ratings
These are editorial judgments on a five-point scale: 1 means substantial documented obstacles or very little supporting evidence; 2 means important requirements or evidence gaps remain; 3 means useful documented support with material limits; 4 requires broad support with few material gaps; 5 requires unusually complete evidence for the category. Unknown adoption and untested compatibility do not earn a high score. The ratings are not benchmark results.
Setup receives 2/5 because successful recorded setup checks coexist with a 3,223.9 MB checkout, Podman requirements and document converters. Documentation receives 3/5 because the supplied README, architecture and methodology explain the workflow and scoring, while our run does not validate the full tutorial. Community receives 2/5 because the extracts show detailed contributor reports and a maintainer-recorded resolution, but do not establish adoption or response-time consistency. Maturity receives 3/5 because recorded software tests completed successfully and the converter issue has a documented later closure, while legal workload validation and task-contract questions remain outside our measured evidence.
Choose the evaluation that answers your question
LegalBench describes legal reasoning tasks as datasets of input-output pairs, including classification, extraction and question answering. Choose it when that task format answers your evaluation question.
HELM describes standardized datasets, a unified provider interface and metrics beyond accuracy. It is relevant to a broader foundation-model evaluation program, with the qualification that its README records maintenance mode from June 1, 2026.
For a LAB trial, start with one task whose documents a lawyer can inspect. Confirm that required facts are available to the agent, verify that deliverables can be read, and review both judges' criterion-level reasoning before expanding the evaluation. That produces a more interpretable result than treating a successful software-test command as evidence of legal performance.