YAML cases turn Skill behavior into a repeatable check
The 450-file skill-up repository gives Agent Skill authors a test harness built around eval.yaml and case files. Each case describes the environment, engine, model, inputs, and judging method. The runner creates a workspace, installs the Skill, invokes the selected agent, grades the result, and writes structured output. This makes a failed behavior reproducible in CI instead of leaving it as a transcript someone has to interpret by eye.
Its roughly 70,817 source lines support rule-based checks, scripts, and an agent_judge. Reports include result.json, Anthropic-compatible grading and benchmark files, JUnit XML, Markdown, and HTML. It can also import Anthropic-style evals.json. Benchmark mode runs cases with and without the Skill, which is useful when a plausible-looking skill makes the underlying agent worse on tasks it already handled.
Four built-in engines expose portability problems early
The 450-file checkout lists Claude Code, Codex, Qoder CLI, and Qwen Code as built-in engines. A custom engine can run as a local command or accept an HTTP request, using a documented session input and result contract. That range lets one suite test whether a Skill depends on one client's tool naming, transcript format, installation path, or model protocol.
Cross-engine judging is not settled in every combination. Open issue 104 reports that an agent_judge inherits the case engine even when its configured model belongs to another provider. The reported Codex case works with an OpenAI-protocol judge and fails with an Anthropic judge. Until that issue is resolved, keep the judge compatible with the evaluation engine or validate the exact pairing before relying on its grade.
What happened when we ran it
Our sandbox installed 150 npm packages in 35 seconds and consumed 95 MB on disk. Npm audit reported 0 known vulnerabilities. The package exposed neither a build script nor a test script, so those steps were skipped. This is an important boundary: the root npm package is the VitePress documentation site, while the evaluator itself is written in Go.
commit 8566175 contained about 70,817 source lines in a 6.2 MB checkout. Our scan found 9 CI workflow files, no Dockerfile, and no tests directory. Those signals describe the repository layout seen by the harness; they do not replace executing the Go test commands or the model-backed end-to-end cases. We did not run either through the npm project, and the review should not imply otherwise.
Our measurement setup was an unprivileged Debian container with 3 CPUs, 8 GB of RAM, Node.js 22, and no secrets. It confirms that the docs dependency tree installs cleanly at the measured commit. It says nothing about the quality of a model judge, the behavior of an external agent CLI, or the availability of a configured OpenSandbox service.
Go 1.25 builds the CLI, while real evals need agent access
The Go module declares Go 1.25. Users can install a release through the provided shell installer, or let the bundled skill-upper Skill guide an agent through installation. A useful run still needs more than the binary: an Agent Skill, realistic cases, the chosen client, model credentials, and judges that can make a clear pass or fail decision. An OpenSandbox runtime adds its own service endpoint and runtime settings.
Configuration has 4 precedence layers: embedded defaults, a user file, a project file, and an explicit path. Existing environment variables win over values supplied by configuration. The docs recommend environment references for secrets, but also state that the current user-config fields do not carry redaction tags. Teams exporting OpenTelemetry data should review what they set and where credentials appear before sharing configs or reports.
The v0.9.1 Linux action pins a separate runner image
The v0.9.1 repository includes a Docker-based GitHub Action for Linux runners. It can select an engine, provider, model, API key, target suite, and parallelism. Its runner image prebuilds the CLI and 3 agent clients, and the production action is supposed to reference that image by immutable digest. A release tag captures the action file present at that commit, but publishing a CLI release does not automatically rebuild the runner image.
That split affects reproducibility. The documentation tells maintainers to synchronize the pinned CLI version and publish a tested image manually. Users who want a fixed setup should choose a post-refresh commit SHA rather than assume a CLI tag contains an image with the same version. The v0.9.1 release included fixes around Codex JSONL, Qoder token handling, Windows PowerShell, judges, and report metrics.
Windows runs the Go 1.25 CLI, but WSL2 is the full-agent path
The Windows guide says Go builds, unit tests, the host runtime, and script judges can run natively. PowerShell and batch judges have direct paths, while shell judges need Git Bash. The WSL bash.exe shim is rejected because its Linux paths do not match the Windows paths generated by skill-up. For a remote OpenSandbox Linux guest, the host communicates over HTTP and avoids the local shell mismatch.
Built-in agent clients have a harder limitation: their Node and nvm bootstrap is Bash-based. The docs recommend WSL2 for full Claude Code, Codex, or Qoder evaluations when the agent CLIs are not already installed. The last push was August 26, 2026, six days after v0.9.1, and GitHub listed 726 stars with 16 open issues and pull requests. Maintenance is current, while the open protocol and Windows work shows where adopters should test first.