mrkeyoor.com_
Tue 01 Sept 17:08 UTC
AI Toolsevaluationupdated 01 Sept 2026

SkillSpector review

SkillSpector checks AI-agent skill files for prompt injection, data theft, dangerous code, excessive permissions, vulnerable dependencies, and other install risks. It reads the skill without running it, then combines static checks with optional language-model analysis to produce a score and recommendation.

trackingstars / 7d
Verdict

Our run installed 103 npm packages in 35 seconds and used 232 MB, but the checkout exposed no build or test target, so it did not verify the Python scanner. Use SkillSpector as pre-install triage alongside coverage checks and human review, especially for skills with scripts or broad permissions. Fail incomplete scans in your own wrapper, choose the data-egress mode deliberately, and keep dynamic testing for behavior the static scanner cannot see.

We ran it

Lab card: what happened when we ran SkillSpectorScreenshot of SkillSpector (docs.nvidia.com/skills/scanning-agent-skills)
Install✓ · 35s103 packages · 232 MB
Buildn/ano build script
Testsn/ano test script
Known vulns00 critical · 0 high · 0 moderate · 0 low (npm audit)
Repo333 files~110,305 lines of source · 5.3 MB · 4 CI workflows · Dockerfile · tests dir

Answers from our run

Does SkillSpector build from source?

Dependencies installed in 35 seconds (103 packages), and the project has no separate build step. We cloned commit 7805bb9 into a clean Debian container with 3 CPUs and no project-specific setup.

Does SkillSpector have tests you can run?

Not through a standard command: the project exposes no test script or target that our harness could run.

Does SkillSpector have known vulnerabilities in its dependencies?

npm audit found none in the dependency tree at the time of our run.

Who should not use SkillSpector?

CI pipelines that treat exit code 0 as proof of a complete scan: issue 389 shows that partial coverage can exit exactly like 100 percent coverage.

What are the alternatives to SkillSpector?

Semgrep, Trivy, Promptfoo. Our run installed 103 npm packages in 35 seconds and used 232 MB, but the checkout exposed no build or test target, so it did not verify the Python scanner.

Setup3/5Python or Docker paths are clear; our npm run found no test target
Docs5/5Rules, scoring, egress, limits, providers, MCP, and CI are explicit
Community5/515,512 stars with active releases and issue handling in September
Maturity3/5Broad scanner, with incomplete-scan and unreachable-rule gaps open

Who it’s for

Teams that install Claude Code, Codex, Gemini, Pi, or MCP skills from outside their organization.
Security engineers who want JSON or SARIF findings in a review or CI workflow.
Skill catalog maintainers who need repeatable scans, baselines, and an allow, warn, or block decision.
Developers who want a local static-only scan before deciding whether deeper review is justified.

Who it’s NOT for

CI pipelines that treat exit code 0 as proof of a complete scan: issue 389 shows that partial coverage can exit exactly like 100 percent coverage.
Teams that require dynamic behavior analysis, image-text inspection, encrypted or compiled-code analysis, or dependable non-English detection: the README lists all four as limitations.
Organizations that need semantic analysis but cannot send eligible skill contents to an LLM provider: the README says the default LLM stage transmits those files to the configured endpoint.
Gates that specifically depend on deterministic trigger-abuse or version-rug-pull rules today: issues 458 and 472 report that those rules read manifest fields real scans do not populate.

Setup reality

Our lab's npm-oriented checkout at commit 7805bb9 installed 103 packages in 35 seconds and used 232 MB. It exposed no build script and no test script, so both steps were skipped. Npm audit reported 0 known vulnerabilities. That run did not install or exercise the documented Python scanner.

The main CLI requires Python 3.12 or newer and a virtual environment, installed with uv or pip; the included Dockerfile avoids a host Python install. MCP use needs the optional mcp extra. Static scans need no model key, while semantic scans need a supported provider, local model endpoint, or authenticated Claude or Codex CLI.

Default semantic analysis sends eligible file contents to the selected provider. OSV checks send dependency names and versions even with --no-llm, with a smaller offline fallback. Remote HTTP MCP has no authentication, so routable deployments need an authenticating reverse proxy.

SkillSpector v2.11.0 scans skill bundles without executing them

SkillSpector v2.11.0 accepts a Git repository, URL, zip, directory, or single file and returns a risk score, severity, recommendation, and findings. Its static stage combines pattern matching, Python AST checks, taint tracking, YARA signatures, permission analysis, and live OSV dependency lookups. An optional LLM stage judges context and intent. Reports can be terminal text, JSON, Markdown, or SARIF. This is a useful pre-install view because the scanner reads suspect content without importing packages or running scripts from the skill.

The source is substantial for a specialist scanner. Our checkout had 333 files, about 110,305 lines of source, and used 5.3 MB before dependencies. The README lists 71 patterns across 17 categories, including prompt injection, exfiltration, privilege escalation, supply-chain behavior, memory poisoning, trigger abuse, MCP permissions, and tool-description poisoning. A score condenses those findings, but the individual rule, file, evidence, confidence, and remediation fields are what a reviewer can investigate. Security teams should retain the detailed report rather than storing only the final recommendation.

A SAFE score still needs coverage and human review

SkillSpector v2.11.0 assigns 0 to 20 points a LOW severity and a SAFE recommendation, while scores above 50 become DO NOT INSTALL. Executable scripts multiply the score. The project also supports baselines that suppress reviewed findings by rule, path, or fingerprint, which helps teams focus on new changes. Static analysis favors recall and can produce false positives; the optional semantic stage aims to filter them. Neither mode proves safety. A missed file, unsupported format, model error, or rule assumption can still leave dangerous behavior outside the report.

Our lab installed 103 npm packages in 35 seconds and recorded 232 MB on disk. That result describes the repository's npm surface, not the documented Python 3.12 CLI. The checkout offered no npm build or test script, so our harness skipped both. Anyone evaluating the scanner should follow the uv, pip, or Docker route and run the Python tests separately. The included Dockerfile gives a contained packaging path, while --no-llm gives a static-only trial without model credentials.

What happened when we ran it

Our npm-oriented run at commit 7805bb9 completed installation in 35 seconds. Npm audit found 0 known vulnerabilities at every listed severity. The repository scan found 4 CI workflow files, a Dockerfile, and a tests directory. Our measurement setup was a fresh unprivileged Debian container with 3 CPUs, 8 GB of RAM, Node 22, and no secrets. These facts show that the npm dependencies installed cleanly; they do not show that a SkillSpector scan, Python package build, or test case succeeded.

There was no build script or target, so the build step was skipped. There was also no test script or target, so our run produced no passing, failing, or skipped test count. The tests directory is a repository signal, not a test result. This distinction matters because release v2.11.0 documents extensive Python validation, but those are NVIDIA's release claims rather than measurements from our sandbox. A buyer should run the Python suite in the same image and dependency mode chosen for the production gate.

Partial coverage can still exit zero

Issue 389 reports that a scan with 86 percent component coverage exits 0 just like a complete scan, because coverage metadata does not affect the CLI code. Issue 460 adds a related limit in v2.11.0: the whole workflow has a 60-second budget that cannot be raised through a flag or environment variable. Its reporter saw large scans return partial results with varying coverage. CI users should parse analysis_completeness, reject incomplete scans, and distinguish a clean finding set from a scanner that did not inspect everything.

Our checkout supplied no npm test target, even though it contained a tests directory and 4 CI workflows. That leaves issue reports and upstream release validation as context, not confirmation from our lab. Issue 458 says three dedicated trigger-abuse rules look for a triggers field absent from supported skill specifications. Issue 472 says the RP3 version rule reads a manifest key the real context builder never populates. Both reports include direct reproductions. Treat rule counts as advertised coverage, then verify that the specific controls your policy depends on can fire end to end.

SkillSpector v2.11.0 sends semantic-scan contents to a provider

SkillSpector v2.11.0 enables LLM analysis by default and sends analyzer-eligible file contents to the configured provider. --no-llm keeps those contents local, although the SC4 dependency check still sends package names and versions to OSV.dev when network access exists. Without OSV access, a small built-in advisory list is used. The MCP server adds another boundary: stdio is suited to local agents, while HTTP ships without authentication. Remote operators must add an authenticating proxy and should keep the documented rejection of local file paths over HTTP.

The project was pushed on September 1, 2026, and GitHub listed 15,512 stars plus 90 combined issues and pull requests. Release v2.11.0 arrived on August 28 with npm lockfile analysis, bundled-hook findings, and parser fixes. Our 103-package npm audit found 0 known vulnerabilities, but we did not run the scanner itself. SkillSpector earns a place before installation because it gives reviewers structured evidence without executing untrusted code. Its recommendation should remain one input to a gate that also checks completeness, provenance, signatures, and runtime behavior.

Alternatives

ProjectWhat it isPick it when
SemgrepA general static-analysis engine with custom rules for source code and configuration.pick this instead when your team wants to own precise code rules across whole repositories rather than use agent-skill-specific scoring.
Trivy gh↗A scanner for vulnerable packages, secrets, misconfiguration, licenses, and infrastructure artifacts.pick this instead when container, filesystem, dependency, and infrastructure coverage matter more than prompt-level skill risks.
Promptfoo gh↗An evaluation and red-team framework for testing prompts, models, and agent applications through execution.pick this instead when runtime behavior and adversarial model responses matter more than pre-install file inspection.

What people are saying

  1. [github-trending] NVIDIA/SkillSpector

Sources

  1. SkillSpector README
  2. SkillSpector v2.11.0 release
  3. Partial coverage exit-code report
  4. Fixed workflow budget report
  5. Trigger-abuse rule reachability report
  6. RP3 manifest version rule report

More ai tools reviews

claudian · robin · mjlab · MoGe · awesome-design-md · Code-as-World · the whole board →