Legal evaluation that looks like work
Most legal AI benchmarks ask a model to classify a passage, recall a rule, or choose an answer. Harvey LAB asks an agent to work through a matter file and produce the artifact a lawyer requested. A corporate M&A task might provide 60 synthetic documents and expect a Word memorandum. Another task may require extracting terms into a structured file, reviewing an agreement, or drafting a package. That change in unit of evaluation makes LAB much more relevant to teams building agents that read and write real office documents.
The repository combines 1,600-plus tasks across 24 legal practice areas and contracting with a Python execution harness. Each task supplies instructions, source documents, named deliverables, and expert-written pass or fail criteria. The project is MIT licensed, tagged v1.0, and includes no database or hosted service. Tasks live on disk, generated runs go under results/, and reports are static HTML. This simple layout makes runs inspectable and easy to archive.
The harness is the real differentiator
LAB does not stop at publishing a dataset. Its harness places an agent in a per-task Podman container with networking disabled, Linux capabilities dropped, source documents mounted read-only, and a writable output area. Six tools let the agent run shell commands, read, write, edit, search, and list files. Parsing support covers Word, Excel, PowerPoint, PDF, and text through tools including Pandoc, MarkItDown, pandas, and pdfplumber.
That environment gives evaluations a consistent operational shape. The harness records the full transcript, token use, wall time, files written, and how many documents the agent read or skipped. Teams can inspect a failure as a trajectory, not merely a bad final answer. Adapters cover Anthropic, OpenAI, Google, Mistral, and Fireworks model routes, so the same task can exercise different providers without rewriting its documents or rubric.
The tutorial is excellent. It walks through environment setup, model configuration, task inspection, a run, grading, report reading, alternative models, sweeps, and comparison dashboards. A first small run is estimated at about 20 minutes. The example is concrete enough that an evaluation engineer should reach a useful report without reverse-engineering the package.
Strict scoring is useful and easy to misuse
Every rubric criterion receives a binary verdict from an LLM judge. A task earns 1.0 only if every criterion passes; one miss produces 0.0. LAB still reports the fraction of criteria passed as a diagnostic. Its rationale is sensible for legal work: a diligence memo that finds most issues but omits one material problem can still fail its purpose.
The problem is that strictness magnifies errors in the benchmark itself. Open issue 115 documented a rubric that mixed merger-guideline thresholds, causing a correct answer to fail the entire task. Recent pull requests have corrected stale entity names, source-matter mismatches, and rubric details. The maintainers are responding quickly, but users should treat task quality as something to audit, not an assumption inherited from the v1.0 tag.
LLM judging creates another layer of uncertainty. The default judge evaluates each criterion semantically against the relevant deliverable, with no golden answer. Optional dual judging runs two named judges and preserves both outputs. That is better than hiding disagreement, yet it does not make verdicts objective. Open issue 114 reports provider-dependent behavior around fuzzy deliverable matching. Serious comparisons should repeat runs, preserve judge versions, inspect borderline reasoning, and avoid ranking models from a single sweep.
Setup is manageable, running it is not free
On macOS or Linux, cloning the repo and running scripts/setup.sh handles much of the initial environment. The project requires Python 3.12, uses uv, and depends on document parsers plus provider SDKs. Podman supplies isolation. Windows users need Windows 11, enabled CPU virtualization, WSL2, and a reboot on the first setup.
A model-provider key is required for the agent, and the default judge needs an Anthropic key. Dual judging also needs an OpenAI key. A single task is easy to budget; a practice-area sweep can multiply agent turns, criterion-level judge calls, output files, and reports rapidly. The comparison tools report latency and estimated cost, which helps, but teams should dry-run a narrow selection before pointing the harness at the full corpus.
Use it as a test suite, not a certification
Project health is strong for such a young repository. Harvey LAB was created in March 2026, had about 650 stars at review time, and was pushed on August 7. GitHub showed 46 open issues and pull requests combined, with new fixes and task corrections active on August 8. There is a v1.0 tag but no GitHub Release entry, so current push and issue activity are better health signals than release cadence.
The right use is an internal regression suite. Select tasks matching your product, have lawyers inspect their source files and criteria, pin the repository revision and judge configuration, then compare repeated runs. LAB offers rare visibility into document handling and missed requirements. It cannot prove that an agent is ready for client work, but it can expose where that agent fails before clients do.