Version 0.122.1 combines evaluation and red teaming
Promptfoo 0.122.1 puts prompt comparison, model evaluation, and adversarial testing behind one command-line workflow. A YAML file defines prompts, providers, inputs, and assertions. The runner can compare hosted APIs, local models such as Ollama, or custom code, then show the results in a browser. That is useful when a team has outgrown screenshots and hand-picked examples but is not ready to build its own evaluation service.
The repository also reaches well beyond a prompt matrix. It includes red-team probes, model-graded and deterministic assertions, caching, shareable reports, and CI output in JSON, HTML, and JUnit XML. Our checkout contained 12 CI workflow files, a Dockerfile, a tests directory, and npm workspaces. That breadth makes Promptfoo credible as a common test layer, although it also explains why working on the source resembles maintaining an application platform rather than installing a tiny developer utility.
Node.js 22.22.0 is the floor, and provider keys are recurring work
Node.js 22.22.0 or newer is required for npm and npx use, while the installation guide recommends Node.js 24 LTS. The shortest trial is npx promptfoo@latest init --example getting-started, followed by an evaluation and the local viewer. Homebrew and a global npm package are supported too. Those paths avoid compiling the repository, which matters because our source build did not complete.
A useful evaluation still needs decisions that an installer cannot make. You choose providers, supply test inputs, and decide which assertions deserve to block a change. Most hosted providers need API credentials, and model-graded checks can add calls beyond the target response. CI users also need to control rate limits, cache behavior, output retention, and spend. Promptfoo records basic usage telemetry by default; its docs say prompts, outputs, test cases, keys, and full configuration files are excluded, and an environment variable disables collection.
What happened when we ran it
Our sandbox installed the promptfoo repository in 126 seconds. npm added 2,415 packages, and the completed dependency tree occupied 3,807 MB on disk. The checkout itself had 5,524 files, roughly 995,274 source lines, and a size of 238 MB. We cloned commit e3b3645 and ran it in an unprivileged Debian container with 3 CPUs, 8 GB of RAM, Node.js 22, and no secrets; that was our measurement setup.
The build exited with code 1 after 79 seconds. Its final lines show the tsdown process ending with SIGTERM, which triggered termination of the other processes. The app task had started tsc -b && vite build and then received SIGTERM as well. The log tail gives no cause for those signals, so it would be speculation to blame memory, the container, or the application compiler. The defensible result is simpler: the full source build failed in our stated environment.
Tests had a much better result. Vitest finished in 269 seconds with 23,460 passing tests, 0 failures, and 10 skipped out of 23,470. That is strong evidence that the tested behavior held at commit e3b3645, even though it does not cancel the separate build failure. A contributor would need both commands green before treating the checkout as release-ready on the same class of machine.
The 30 audit findings need review before CI adoption
npm audit reported 30 vulnerabilities in the installed tree: 27 high and 3 moderate, with 0 critical and 0 low findings. The measurement does not tell us whether each advisory is reachable in normal Promptfoo use, a development-only package, or a false fit for this application. It does give security teams a concrete review queue before credentials and evaluation data are placed in a shared CI runner.
The scale cuts both ways. Passing 23,460 tests shows far more internal checking than a young prompt script usually has, while 2,415 installed packages create a wide dependency surface. The README's global install makes a first evaluation look small because users do not see the source toolchain. Teams that only run the published CLI can start there, but anyone embedding the library or contributing code should reproduce the audit and build on the exact deployment image they plan to trust.
An August 27 push and 528 open items show active maintenance
GitHub recorded the latest push on August 27, 2026, one day after release 0.122.1. The repository had 24,603 stars and 528 combined open issues and pull requests when fetched. That combined count is not a defect total. Recent activity includes release work, provider fixes, tracing changes, and red-team token accounting, which points to a maintained project with a busy change stream rather than an abandoned package.
Release 0.122.1 touched provider support, tracing, dependency handling, assertions, and red-team usage accounting. A team adopting Promptfoo should pin the CLI version in CI and review changes before upgrading, especially when provider behavior or token accounting affects budgets. The active queue means bugs can get attention, but it also means a moving interface around many external APIs. Current maintenance is good; quiet upgrades are still a poor idea for a test gate.
A 3,807 MB install is justified only when both jobs matter
Promptfoo makes the most sense when one team owns ordinary LLM evaluation and security probing. DeepEval is the cleaner comparison for Python teams that want tests to feel like the rest of a pytest suite. NVIDIA's Garak is narrower when the assignment is vulnerability scanning. Langfuse belongs on the shortlist when production traces, shared datasets, and experiment history are more important than a local-first runner.
Our 126-second install and 269-second test run make Promptfoo practical to trial, while the failed build and 30 audit findings keep it out of the effortless category. I would use it for a team with several providers, a real regression corpus, and someone assigned to maintain the evaluation definitions. For a single prompt, one model, and occasional manual review, the 2,415-package source tree asks for more ownership than the problem warrants.