Garak separates attacks, targets, and verdicts
Garak v0.17.0 treats a scan as three distinct parts. A generator talks to the target model, a probe supplies adversarial interactions, and one or more detectors judge the replies. That separation lets the same prompt-injection probe run against OpenAI, Bedrock, Ollama, a Hugging Face model, or a custom REST endpoint. It also lets a team add one detector without writing a new transport. The design is closer to a security scanner than a model leaderboard.
The included families cover data leakage, encoded prompt injection, jailbreaks, malicious package suggestions, system-prompt extraction, misinformation, and unsafe content. Some probes are static; others assemble, vary, or adapt prompts in response to the target. Results are expressed as pass and attack-success rates for each probe and detector pairing. Garak's own FAQ warns that those scores are not normalized and cannot support meaningful comparisons between unrelated probes.
Ten generations per prompt can turn one command into a large bill
A default scan selects every known probe and makes 10 generations per prompt, according to the README. That breadth is useful for discovery and risky for a metered API. The FAQ says all-probe runs can take hours and recommends --config fast, selected probes, or parallel attempts for remote models. Start with a target-specific threat list, record the exact model settings, and expand only after estimating the request count. Local models replace API cost with RAM, GPU, and model-download requirements.
What happened when we ran it
Our sandbox installed 240 packages in 100 seconds and consumed 6,435 MB on disk. The source checkout was only 11.8 MB, holding 873 files and roughly 80,571 lines, so nearly all of that footprint arrived with the Python environment. The build then succeeded in 5 seconds. Pip-audit found 2 known vulnerabilities. Those counts describe commit 3f50ea5 in a 3-CPU, 8 GB Debian container, without credentials or a target model.
The test process did not finish within 900 seconds. Its final visible progress reached 19%. An earlier line at 8% began with F, and several later lines contained skip markers. Pytest never printed a final summary in the supplied tail, so we cannot state the total passed, failed, or skipped, or identify the failing case. The repository has a tests directory and 14 CI workflow files, but our full local result remains a timeout with at least one visible failure marker.
JSONL reports preserve evidence and sensitive model output
Every run writes a JSONL report containing model parameters, prompts, responses, and evaluation scores. Garak also keeps a continuing debug log and a hit log for attempts that triggered detectors. That evidence is what makes a result reviewable: an analyst can see whether a detector found a real weakness, misunderstood the output, or used an unsuitable prompt. The same files can contain leaked-looking credentials, generated malware text, slurs, or proprietary target responses. Store them like security-test artifacts.
The analysis tools can group findings under taxonomies such as the OWASP Top 10 for LLM applications, and v0.17.0 added EU AI Act reference tags. Mapping a probe to a taxonomy helps sort work; it does not turn an attack-success rate into regulatory compliance. The FAQ explicitly says the raw probe scores lack scientific validity. Use the tags for report navigation, then attach reproduced evidence, target configuration, and human severity judgment to any ticket or release decision.
Open detector issues rule out unattended pass or fail gates
Issue 2172 demonstrates a concrete false-negative path at commit 3f50ea5: one safe MongoDB placeholder can mask a separate AWS-style credential match in the same output. Issue 2136 describes the opposite direction in v0.16.0, where an unparseable judge answer could be recorded as a successful attack. The v0.17.0 release contains several judge-shape fixes, but issue 2136 remained open when fetched. Detector output deserves inspection on both sides of the pass line.
Conversation-level testing also has a stated limit. Issue 2173 says even multi-turn probes hand individual attempts to detectors, with no trajectory-aware score for escalation across a session. Issue 2141 reports generator retry paths that can continue indefinitely after connection errors; v0.17.0 fixed one unreachable OpenAI-compatible-target hang, while the broader retry issue stayed open. Set external timeouts and choose a framework built around conversations if gradual multi-turn compromise is the primary threat.
Version 0.17.0 is active and still labeled alpha
GitHub recorded 9,195 stars, 421 combined issues and pull requests, and a last push on September 9, 2026. Release v0.17.0 shipped the same day with Python 3.13 support, removal of Python 3.10, detector fixes, Ollama changes, and EU AI Act mapping. New issues and pull requests were active on September 10 and 11. The large queue includes plugin proposals and contributions as well as bugs, so 421 is not a defect count.
The package classifiers still say Development Status 3, Alpha. That label fits the fast-changing threat coverage and recent detector work better than the NVIDIA name might suggest. Garak has strong documentation, cross-platform CI, an Apache-2.0 license, and a visible release cadence. A same-week release plus active triage shows healthy maintenance. It does not erase our 900-second test timeout or remove the need to pin versions when scan results feed security decisions.
Garak is a discovery tool for analysts, not a compliance stamp
Garak earns a trial when the goal is to expose an authorized model to many known attack styles and preserve the evidence. The price is a 6,435 MB environment, potentially long or costly scans, and detector behavior that requires an analyst. Select probes from the system's threat model, cap requests and runtime, secure the JSONL output, reproduce serious hits, and compare the same probe configuration after model or guard changes. That workflow turns Garak's breadth into useful engineering work.