mrkeyoor.com_
Tue 01 Sept 17:45 UTC
AI Toolsevaluationupdated 26 Aug 2026

AI-Infra-Guard review

AI-Infra-Guard is an English-documented platform for testing AI services, agents, Model Context Protocol servers, agent skills, and language models; its README links eight translations. It puts live-service vulnerability checks, source inspection, jailbreak exercises, and model API checks in one self-hosted web app.

+208stars / 7d
Verdict

Our AI-Infra-Guard run installed 301 packages and built successfully, but 7 of 16 tested packages failed, so it belongs in an evaluation lab before it belongs in a security process. Use it when one isolated team needs its six scanning and evaluation jobs and can verify every finding. Choose a narrower tool when model probing, prompt tests, or model-file inspection is the only requirement.

We ran it

Lab card: what happened when we ran AI-Infra-GuardScreenshot of AI-Infra-Guard (tencent.github.io/AI-Infra-Guard)
Install✓ · 30s301 packages
Build✓ · 69s
Tests✗ · 147s9 passed · 7 failed of 16 (go test)
Repo5867 files~139,854 lines of source · 138.2 MB · 3 CI workflows · Dockerfile

Answers from our run

Does AI-Infra-Guard build from source?

Dependencies installed in 30 seconds (301 packages), and the build succeeded in 69 seconds. We cloned commit 4908db1 into a clean Debian container with 3 CPUs and no project-specific setup.

Do AI-Infra-Guard's tests pass?

Not all of them: 9 of 16 passed and 7 failed when we ran the project's own test command (go test). Some failures need services or credentials a bare container does not have.

Who should not use AI-Infra-Guard?

Anyone planning a public deployment: the README says the web platform has no authentication and must stay off public networks.

What are the alternatives to AI-Infra-Guard?

Promptfoo, Garak, ModelScan. Our AI-Infra-Guard run installed 301 packages and built successfully, but 7 of 16 tested packages failed, so it belongs in an evaluation lab before it belongs in a security process.

Setup2/5Build passed, but 7 of 16 test packages failed
Docs4/5Clear Docker, scan-target, API, and credential guidance
Community4/5Pushed August 26 with 28 open issues and PRs
Maturity3/5v4.6.0 is active, but isolation and test failures need care

Who it’s for

Security teams that need one internal workbench for AI infrastructure, MCP, skill, agent, and model testing.
Developers who want a standalone skill scanner that can emit JSON for CI.
Authorized red teams testing running vLLM, Ollama, ComfyUI, n8n, or similar services.
Operators willing to isolate a scanner that processes hostile code and connects to target systems.

Who it’s NOT for

Anyone planning a public deployment: the README says the web platform has no authentication and must stay off public networks.
Operators who cannot isolate a privileged container: the supplied Compose file gives the agent SYS_ADMIN and disables its default seccomp profile.
Teams that require a clean upstream test run before evaluation: our commit passed 9 of 16 tested packages and failed 7.
Air-gapped teams expecting every mode to work without another model: skill scans and jailbreak evaluations require an LLM endpoint and API key.
Long-running jailbreak campaigns that must resume after interruption: issue 537 says recovery is unavailable, and the request remains open.

Setup reality

Our sandbox installed 301 packages in 30 seconds and built commit 4908db1 in 69 seconds. Tests failed after 147 seconds: 9 of 16 packages passed and 7 failed. The log tail shows TestReadSmallFile trying to open /mcp-server/src/mcp_server/app_config.py, which did not exist at that path.

The documented web route needs Docker 20.10 or newer, at least 4 GB of RAM, 10 GB of disk, and Docker Compose. LLM-backed skill scans and jailbreak tests also need a model endpoint and API key; infrastructure scans need network access to the authorized target service.

The web app has no authentication. Its published Compose file maps port 8088, uses latest image tags, and grants the agent SYS_ADMIN with seccomp disabled. Run it on a dedicated internal host, restrict ingress, pin images, and protect uploaded code, reports, and model credentials.

Six security jobs share one internal web app

AI-Infra-Guard groups 6 jobs in one project: infrastructure fingerprinting, OpenClaw checks, agent exercises, MCP and skill inspection, jailbreak evaluation, and model API relay auditing. The infrastructure scanner connects to a running service such as vLLM, Ollama, or ComfyUI and matches the detected component against vulnerability rules. MCP servers and skills can instead be supplied as a remote repository or uploaded source archive.

That breadth is useful for a security team whose AI estate has spread beyond prompts. The same interface can examine a service version, inspect tool code, and send adversarial cases to a configured model. The README says the rule library covers more than 100 AI components and 2,000 CVEs. Those are Tencent's coverage figures. A match still needs confirmation against the deployed package and any vendor patches.

What happened when we ran it

Our sandbox cloned commit 4908db1, a 138.2 MB checkout containing 5,867 files and about 139,854 lines of source. Installation succeeded in 30 seconds with 301 packages. The build also succeeded, taking 69 seconds. The repository had 3 CI workflow files, a Dockerfile, and a Compose file, but no separate tests directory.

The test command ended with exit code 1 after 147 seconds. Go reported 9 passed packages and 7 failed packages out of 16. In the final failure shown to us, TestReadSmallFile attempted to read /mcp-server/src/mcp_server/app_config.py and received a file-not-found error. The log establishes the missing path and nothing more, so we cannot say whether the fault sits in packaging, the test fixture, or our container layout.

A successful 69-second build means the source compiled in the stated Debian sandbox. The 7 failed packages mean the full check did not clear the same environment. Before trusting scanner output, rerun the tests in the deployment image and inspect every failure. A security tool deserves a stricter acceptance bar because a silent miss can look exactly like a clean target.

The 4 GB quick start leaves security work for the operator

The main quick start asks for Docker 20.10 or newer, at least 4 GB of RAM, and 10 GB of disk. It clones the repository, starts prebuilt images with Docker Compose, and exposes the interface on port 8088. Building from source is another Compose command. A separate aig-skill-scan Python package is available when the web platform is unnecessary.

Credentials depend on the job. Jailbreak evaluation needs a target model base URL and API key. The skill scanner also expects an LLM key and model choice. Infrastructure scanning needs access to a live, authorized service address, while MCP and skill inspection can accept source. Uploaded code, model credentials, scan reports, and logs should all be treated as sensitive assessment data.

Port 8088 has no authentication and the agent is privileged

AI-Infra-Guard's README explicitly says the platform lacks authentication and should not be deployed on public networks. The current image Compose file maps 8088:8088, so access control is the operator's job. Put the app behind a restricted internal route or bind it locally, and keep the scanner away from production control planes unless the test has written authorization.

The agent container receives SYS_ADMIN, and its seccomp profile is set to unconfined. Both web and agent images use the moving latest tag. Those choices make a disposable VM or dedicated worker a better home than a shared application host. Pin a tested release, inspect the Compose changes on upgrade, limit target reach, and destroy transient scanning environments after sensitive work finishes.

Version 4.6.0 fixes bypasses, while interrupted runs remain open

Release v4.6.0 shipped on August 26, 2026, with an API checker, LLM API poisoning detection, stricter rule validation, MCP SDK fixes, and Docker compatibility work. It also includes the earlier response to issue 531, where compiled Python bytecode could be hidden from the skill scanner and receive a safe result. The issue is now closed, but it is a useful regression case to keep locally.

Issue 537 was still open on August 26 and describes jailbreak runs that cannot recover after interruption. That matters when a campaign combines many methods and samples. Start with a small set, confirm that results and provider usage are recorded, then increase the matrix. A scanning dashboard can make a large job easy to launch without making it cheap or recoverable.

Same-day code and issue activity support a trial, not blind trust

GitHub recorded a push on August 26, 2026, the day v4.6.0 was released. The repository had 5,984 stars and 28 open issues and pull requests combined when fetched. Several pull requests were updated that day, including MCP error reporting, Docker work, and API checker changes. That is active maintenance, though the combined count should not be read as 28 bugs.

AI-Infra-Guard earns a trial when its 6 jobs would replace several disconnected checks. Our failed 147-second test run keeps the recommendation qualified, as do the privileged agent and unauthenticated interface. Run it inside a controlled lab, preserve a few known-bad fixtures, and have a person reproduce high-impact findings. If the task is only prompt red teaming or model-file inspection, Promptfoo, Garak, or ModelScan asks for less operational trust.

Alternatives

ProjectWhat it isPick it when
Promptfoo gh↗A developer-oriented framework for LLM evaluations, red teaming, and CI checks.pick this instead when prompt, agent, or RAG testing is the main job and infrastructure scanning is out of scope.
GarakA command-line scanner that probes language models for known failure classes.pick this instead when model behavior is the target and you want a narrower command-line tool.
ModelScanA scanner for unsafe code and serialization risks in machine-learning model files.pick this instead when model artifacts entering a pipeline are the risk, rather than live services, MCP servers, or agent skills.

What people are saying

  1. [github-trending] Tencent/AI-Infra-Guard

Sources

  1. AI-Infra-Guard README
  2. AI-Infra-Guard v4.6.0 release
  3. AI-Infra-Guard repository facts
  4. Published Docker Compose configuration
  5. Compiled Python bytecode scanner bypass report
  6. Interrupted jailbreak scan recovery request

More ai tools reviews

claudian · SkillSpector · robin · mjlab · MoGe · awesome-design-md · the whole board →