mrkeyoor.com_
Fri 16 Oct 18:55 UTC
AI Toolsevaluationupdated 04 Oct 2026

AI-Infra-Guard review

AI-Infra-Guard is a self-hosted security workbench for AI services, agents, Model Context Protocol servers, agent skills, and language models. It scans running services for known vulnerabilities, inspects source, runs jailbreak evaluations, and checks whether a model API is serving what it claims.

+61stars / 7d
Verdict

Our AI-Infra-Guard run installed 301 packages and built in 103 seconds, but 15 of 33 tests failed, so we would keep it in an isolated evaluation lab rather than make it a release gate. Use it when one security team needs its broad set of AI checks and can reproduce important findings. Choose a narrower tool when prompt testing, model probing, or agent-source inspection is the whole assignment.

We ran it

Lab card: what happened when we ran AI-Infra-GuardScreenshot of AI-Infra-Guard (tencent.github.io/AI-Infra-Guard)
Install✓ · 41s301 packages
Build✓ · 103s
Tests✗ · 26s18 passed · 15 failed of 33 (go test)
Repo6410 files~153,457 lines of source · 140.1 MB · 3 CI workflows · Dockerfile

Answers from our run

Does AI-Infra-Guard build from source?

Dependencies installed in 41 seconds (301 packages), and the build succeeded in 103 seconds. We cloned commit 14beb9b into a clean Debian container with 3 CPUs and no project-specific setup.

Do AI-Infra-Guard's tests pass?

Not all of them: 18 of 33 passed and 15 failed when we ran the project's own test command (go test). Some failures need services or credentials a bare container does not have.

Who should not use AI-Infra-Guard?

Anyone planning a public deployment: the README says the web platform has no authentication and should stay off public networks.

What are the alternatives to AI-Infra-Guard?

Promptfoo, Garak, Agent Scan. Our AI-Infra-Guard run installed 301 packages and built in 103 seconds, but 15 of 33 tests failed, so we would keep it in an isolated evaluation lab rather than make it a release gate.

Setup2/5Build passed, but 15 of 33 tests failed and Docker needs 10 GB
Docs4/5Clear targets, Docker limits, credentials, API, and warnings
Community5/5October 3 push, v4.6.4 release, and 26 open pull requests
Maturity3/5Active v4.6.4 project, but no auth and our suite failed

Who it’s for

Security teams that need one internal console for AI infrastructure, MCP, skill, agent, and model testing.
Developers who want a standalone agent-skill scanner with JSON output for CI.
Authorized red teams assessing live vLLM, Ollama, ComfyUI, n8n, or similar services.
Operators prepared to isolate a scanner that accepts source archives and model credentials.

Who it’s NOT for

Anyone planning a public deployment: the README says the web platform has no authentication and should stay off public networks.
Small hosts: the documented Docker route requires at least 4 GB of RAM and 10 GB of disk.
Teams that require a green test gate before adoption: 15 of 33 tests failed in our sandbox run.
Air-gapped teams expecting every mode to work alone: the skill scanner and jailbreak evaluation need an LLM endpoint and API key.
Teams trying to scan repository code for deployed infrastructure flaws: that mode expects the network address of a running AI service.

Setup reality

Our sandbox installed 301 packages in 41 seconds and built commit 14beb9b in 103 seconds. Tests failed with exit code 1 after 26 seconds: 18 passed and 15 failed out of 33. The log tail names failures in TestGrepDirectory, TestReadBigFile, and TestReadSmallFile under internal/mcp/utils, but it does not show their cause.

The Docker route requires version 20.10 or newer, at least 4 GB of RAM, 10 GB of disk, and Compose. Skill scans need LLM_API_KEY. Jailbreak evaluation needs a model base URL and API key. Infrastructure scans need network access to a running target.

The web app has no authentication and should remain on a restricted internal network. Our 140.1 MB checkout contained 6,410 files and about 153,457 source lines, so source builds and review are more involved than the short Compose quick start suggests.

Six AI security jobs share one internal console

A.I.G v4.6.4 puts live-service vulnerability checks, OpenClaw scanning, agent exercises, MCP and skill inspection, jailbreak evaluation, and model-relay auditing in one project. The modes take different inputs. Infrastructure scanning connects to a running service such as vLLM, Ollama, or ComfyUI. MCP servers and skills can arrive as a remote repository or uploaded source archive. Jailbreak work sends a selected dataset to a configured model endpoint.

That breadth is the reason to consider A.I.G. A security team can examine a deployed service, review tool code, and test a model without assembling a separate interface for each job. The README says its vulnerability library covers more than 100 AI components and 2,000 known CVEs. The project supplies those coverage figures. Our sandbox did not test them. Treat a match as a lead to verify against the installed version and vendor guidance.

What happened when we ran it

Our sandbox installed 301 packages in 41 seconds, then built commit 14beb9b successfully in 103 seconds. Our measurement setup was a fresh unprivileged Debian container with 3 CPUs, 8 GB of RAM, Go 1.24, and no secrets. The checkout contained 6,410 files, about 153,457 lines of source, and occupied 140.1 MB. We found 3 CI workflow files, a Dockerfile, and a Compose file, but no tests directory.

The test step failed with exit code 1 after 26 seconds. Go reported 18 passed and 15 failed out of 33. The last lines name TestGrepDirectory, TestReadBigFile, and TestReadSmallFile in internal/mcp/utils. The package finished in 0.086 seconds and failed. The supplied tail does not include the assertion text or an error message that establishes why those cases failed, so we will not assign a cause.

A successful build shows that this commit compiled in the stated container. It does not cancel 15 failed tests, and the failed suite does not prove that the scanner's findings are wrong. The practical response is to run known-safe and known-bad fixtures before using A.I.G in a security process. If those fixtures change result after an upgrade, stop there and inspect the release before scanning production systems.

Docker needs 4 GB of RAM and 10 GB of disk

The documented quick start asks for Docker 20.10 or newer, at least 4 GB of RAM, 10 GB of disk, and Compose. It starts prebuilt images and serves the interface at http://localhost:8088. You can also build the images from source. That is a short path to a first screen, though the 140.1 MB checkout and 103-second build show there is a sizeable system behind it.

Credentials depend on the scan. The standalone skill tool expects LLM_API_KEY and a model choice. Jailbreak evaluation asks for the target model's base URL and API key. Infrastructure scanning needs a live service address, while MCP and skill inspection can work from source. Budget provider usage before launching a large evaluation, and use test credentials that cannot reach unrelated services.

No authentication keeps port 8088 off the public internet

A.I.G v4.6.4 has no authentication, and its README says the platform should stay off public networks. That warning changes the deployment decision. The app can hold uploaded source, model credentials, target addresses, and scan results, so a public 8088 listener would expose more than a harmless dashboard. Keep it on a restricted internal network or a disposable lab host with access limited to the people running the assessment.

Isolation matters for another reason: a security scanner is built to inspect suspicious inputs and connect to targets. Give it only the network reach required for the current job. A team evaluating 1 MCP repository does not need the scanner to see every production subnet. Delete uploaded material and rotate temporary model credentials when the engagement ends. These controls are operator work because the application does not supply a login boundary.

Version 4.6.4 arrived on October 1, 2026

GitHub showed a push on October 3, 2026, two days after v4.6.4 was published. The repository had 6,746 stars and 40 open issues and pull requests when checked on October 4. The API list split that queue into 14 issues and 26 pull requests. Those dates and the active review queue support continued maintenance rather than a project coasting on an old release.

The latest release was small and specific: it added a prompt-injection research toolkit under the research directory and fixed forwarding of optional reasoning effort to a model API. Recent README entries also describe frequent changes to skill, MCP, jailbreak, relay, and CVE rules. Pin the exact release you evaluate, rerun your fixtures, and review rule changes before replacing a working scanner image. Fast rule updates are useful only when your results remain explainable.

Narrower scanners cost less attention for narrower jobs

For a 1-model job, NVIDIA's Garak concentrates on probing the endpoint. Promptfoo is the better fit when prompts, agents, or RAG behavior need repeatable evaluation in CI. Snyk's Agent Scan focuses on agents, MCP servers, and skills. All 3 leave out parts of A.I.G's scope, which is the point: fewer modes mean fewer credentials, targets, and result types to manage.

Choose AI-Infra-Guard when one isolated security team will use several of its modes and has time to verify findings. The 15 failed tests were the stop sign after a workable 41-second install and 103-second build. A.I.G can be a useful workbench, but our measured commit did not earn the trust required for an unattended release gate.

Alternatives

ProjectWhat it isPick it when
Promptfoo gh↗A developer-focused framework for prompt, agent, and RAG evaluation and red teaming.pick this instead when behavior testing and CI checks matter more than scanning deployed AI infrastructure.
Garak gh↗A command-line scanner that probes language models for known failure classes.pick this instead when the model endpoint is the target and you want a narrower CLI.
Agent ScanA focused scanner for AI agents, MCP servers, and agent skills.pick this instead when agent and MCP source checks are the job and you do not need A.I.G's web platform.

Sources

  1. AI-Infra-Guard README
  2. AI-Infra-Guard repository facts
  3. AI-Infra-Guard v4.6.4 release
  4. AI-Infra-Guard open issue activity

More ai tools reviews

universal-modder · DeepEP-Ascend · DeepGEMM-Ascend · hypoarena · jeeves · faster-whisper · the whole board →