How we test: the lab behind every review

Why we run things instead of reading about them

Most software reviews are paraphrased documentation. That is cheap to produce and it tells you nothing a search result does not. The number that matters to a developer is different: does npm install finish, how much comes with it, do the tests pass, how big is the bundle that lands in a browser, and how many known vulnerabilities ride along. Those numbers cannot be read off a README. So we measure them, on our own machine, and print them on the page under We ran it or We installed it. The prose is written from those numbers and is not allowed to invent any others.

The sandbox

Every run happens on a dedicated lab server, not on the machine that serves this site, inside a throwaway Docker container that is destroyed afterwards. The container runs as an unprivileged user with all Linux capabilities dropped and no-new-privileges set, with no credentials of any kind inside it. The limits are the same for every project:

CPU / memory	3 CPUs and 8 GB RAM per container (12 GB for Rust, whose test binaries link large); 768 processes max
Images	Debian Bookworm: node:22 (npm, pnpm, yarn, bun), Python 3.12 with uv, Go 1.24, Rust stable
Network	on, so installs and network-dependent tests work; the container holds no secrets, and we never pass any
Time limits	3 minutes to clone, then 15 minutes each for install, build and tests; a library install gets 10 minutes
Size	repositories larger than 1.2 GB on GitHub are not cloned and are reviewed from their docs, which the page says
Cache	none: every run starts from an empty npm/uv/cargo/go cache, so install times are cold-start times

What we measure for a repository

Clone the default branch at a specific commit (shallow). The commit hash and date are printed on the page.
Detect the ecosystem from its manifest (package.json, pyproject/requirements, go.mod, Cargo.toml), including monorepos whose project lives one or two directories down. A repo with none of these (a curated list, a book, a pure Dockerfile project) is marked not runnable.
Install with the project's own package manager (lockfile respected when present), build with its own build script or go build ./... && go vet / cargo build, and run the tests with its own test script, pytest, go test ./... or cargo test. We record success or failure, the exit code, the wall-clock seconds, and the parsed test counts (jest, vitest, mocha, node:test, bun test, pytest, go test, cargo test).
Dependencies and footprint: how many packages ended up on disk and how many megabytes, plus npm audit or pip-audit for known vulnerabilities where the ecosystem supports it.
Shape: files, lines of source, CI workflow files, Dockerfile and compose files, a tests directory, monorepo workspaces.

What we measure for a library

Install the package into an empty project (npm install or uv pip install): seconds, megabytes on disk, how many packages came along, deprecation and peer-dependency warnings, and whether a native compile step ran.
Import: for npm, both require() and ESM import, which catches ESM-only and CJS-only packages; for Python, import <top-level module> and how long it took.
Types: bundled TypeScript types or @types; py.typed for Python. Module format and the presence of an exports map.
Browser bundle: import * from the package bundled by esbuild for the browser, minified and gzipped. If esbuild cannot build it for the browser, the page says so; that usually means Node-only code.
Vulnerabilities via npm audit / pip-audit, and licence, engines and Requires-Python from the package metadata.

MCP servers go through a separate harness (boot over stdio, initialize, tools/list, the token cost of that list, a 90-second timeout, never a side-effect tool). It is described on each MCP server page.

What a status means

ran	clone and install succeeded; build and tests ran (each may have passed or failed, and the page shows which)
install failed	the project's own install command exited non-zero in a clean Debian container; often missing system packages the README does not mention
build failed	install worked, the build script or compiler did not; the last lines of the log are quoted on the page
tests timed out	the test run did not finish in 15 minutes; we report it as unfinished, not as failed
not runnable	no supported ecosystem manifest at the root or one level down (books, lists, Dockerfile-only projects)
too big / clone failed	we did not run it; the review relies on the project's documentation and says so

What we do not measure, and other limits

One run, one machine. A number is what happened once on our hardware on that date. We re-run projects roughly every five weeks and the page shows the date and commit; we do not average runs.
Not performance. Install seconds and bundle bytes are footprint, not throughput. We do not benchmark runtime speed.
Not a security review. npm audit and pip-audit report known advisories in the dependency tree, nothing more. A zero is not a clean bill of health.
Not test quality. “426 of 447 passed” says the suite mostly passes in our container; it says nothing about what the suite covers. A failure in our container can be the project's fault or ours (a missing system library, a test that needs a database we did not start). We quote the log tail so you can judge.
Sandbox bias. A fresh Debian container with no system packages is harsher than a developer laptop. Projects that assume make, pkg-config, a GPU or a running service will fail here even if they work for you. We say so where we can tell.

How the numbers reach the prose

The writer receives the measurement as a block of facts and a rule: those are the only numbers it may state about install, build, tests, dependencies or vulnerabilities. A validator rejects a review that invents a figure, that reads like boilerplate, or that leans on stock phrasing. Every page prints the measured block as data, separately from the prose, so you never have to take the sentence's word for it. Nobody pays to be measured, and nothing changes a result except re-running it.

Questions about a specific run, or a project you think we got wrong: write to the desk. If a re-run disagrees with the page, the page gets the re-run.