LAB evaluates finished legal work, not isolated answers
Harvey LAB asks an agent to work through a synthetic matter file and produce the kind of deliverable a lawyer would review: a memo, agreement, spreadsheet, or research output. Each task defines instructions, source documents, expected filenames, and a pass-or-fail rubric. The harness saves the agent transcript, tool use, token counts, document coverage, and outputs. An evaluator then grades each criterion against only the deliverables assigned to it and produces a static report.
The dataset dominates the repository. Our commit 7be41d5 checkout contained 63,074 files, about 14,042 source lines, and occupied 3,260.2 MB. That ratio makes sense for a benchmark made of Word files, spreadsheets, PDFs, and task JSON rather than mostly code. The README says the documents are generated synthetically in batches under lawyer guidance and review, and warns that they contain imperfections. No confidential client material should be added by contributors.
All-pass scoring treats one missed issue as a failed task
Every rubric criterion receives a binary judge verdict. A task scores 1.0 only if all criteria pass; there is no partial credit at task or criterion level. The project still reports pooled and per-task criterion pass rates as diagnostics. This choice fits legal review where one omitted material risk can defeat an otherwise polished memo, but it makes each rubric sentence unusually powerful. One faulty criterion can force every correct agent run to fail.
Standard scoring uses 2 independent judges, Claude Sonnet 4.6 and GPT-5.5, and writes the combined artifact only after both finish. Each criterion becomes a separate model call with its relevant output and match standard. That can be expensive across a large task sweep. It also adds judge variance to agent variance. Teams should preserve both judges' files, review disagreements, and repeat a useful subset before interpreting a model difference as stable.
What happened when we ran it
Our sandbox installed 35 Python packages in 37 seconds and used 37 MB. The build check passed in 13 seconds. Pytest ran for 37 seconds and ended with exit code 1: 12,588 tests passed, 23 failed, 59 were skipped, and 39 collection or setup errors were recorded out of 12,650. Pip-audit reported 0 known vulnerabilities.
The final log lines repeatedly showed ModuleNotFoundError: No module named 'anthropic' while importing evaluation and pipeline test paths. That establishes a missing module in our fresh Python 3.12 environment; it does not explain every failure elsewhere in the suite. The checkout had 2 CI workflow files and a tests directory, but no Dockerfile in the repository scan. Our no-secret run did not call an agent provider or either default judge.
Six sandbox tools keep task documents away from the host
Agents receive bash, read, write, edit, glob, and grep. All 6 route through a per-task Podman container with networking disabled, Linux capabilities dropped, source documents mounted read-only, and a writable output layer. That design limits a malicious document or reckless shell command during an agent run. Provider calls happen through the harness, while file parsing occurs inside the same sandbox interface.
The operational setup is larger than the 37 MB Python environment. Podman must work, and document extraction depends on tools such as Pandoc, pdfplumber, pandas, and spreadsheet readers. Windows uses WSL2 and needs Windows 11 plus CPU virtualization. Standard evaluation requires both Anthropic and OpenAI credentials, while benchmarking Google, Mistral, or Fireworks models needs the corresponding provider access. Results remain filesystem-first under ignored run directories, with no database or web service to operate.
Some task contracts currently grade facts the agent never received
Issue #152 identifies 5 of 49 criteria that refer to a $31 million escrow in a litigation-risk task. The report says that figure appears in neither the 9 supplied documents nor the instructions, and 3 criteria require it directly. Issue #149 finds a meeting date present only in a rubric; across 17 scored runs, that criterion passed 0 times. These are benchmark defects, not evidence that every task is unreliable. They still require users to audit task inputs before trusting aggregate scores.
Issue #146 finds count and allowlist inconsistencies in 7 firm-knowledge tasks at the exact 7be41d5 commit we ran. One criterion states 4 of 7 deals while another enumerates 8 members of the population. The report does not claim every extra item is factually wrong; it shows that the executable scoring contract cannot express whether those items are required, optional, negative, or excluded. Strict all-pass grading amplifies that ambiguity.
An unreadable DOCX can be scored as bad legal work
Issue #145 reports that when Pandoc is missing, the evaluator converts the read error into a string and gives that string to the judge as if it were the deliverable. The same run scored 7 of 50 criteria without Pandoc and 34 of 50 after installation. Because the report contains plausible failure reasoning, a user may blame the agent rather than the converter. Preflight every required file type and treat any extraction error as an invalid evaluation.
GitHub showed 1,262 stars, an MIT license, and 56 combined issues and pull requests. The repository was pushed on August 26, 2026, although GitHub returned no latest release object; the README cites tag v1.0. Current issue activity is a health signal because contributors are checking source facts, evaluator behavior, and provider adapters. LAB is useful precisely because it exposes full work products and reasoning traces, but its results need the same legal review discipline it is trying to measure.