Agent Skills need tests, not good intentions
A SKILL.md file can look perfectly sensible and still fail when a real agent interprets it. Instructions may trigger at the wrong moment, conflict with tool behavior, miss a workspace assumption, or produce an answer that appears plausible while skipping a required step. Skill Up treats that problem like software testing. Developers describe cases in YAML, run them through supported agent clients, grade the outcomes, and retain structured results for comparison and CI.
The project supports Codex, Claude Code, Qoder CLI, and Qwen Code as built-in engines, plus custom local agents. A case can be judged by straightforward rules, a script, or another agent. Reports include JSON, Markdown, HTML, JUnit XML, and Anthropic-compatible grading and benchmark files. That range matters because a local author wants readable failure evidence, while a pull-request job needs a machine-readable exit condition.
Skill Up also runs with and without the target skill, following the evaluation pattern documented by the Agent Skills specification. That baseline is essential. A passing answer proves little if the underlying agent succeeds just as often without the skill. The useful question is whether installing the skill improves the behavior the author intended.
The evolution loop is the distinctive part
The repository includes skill-upper, an Agent Skill that drives the evaluator through conversation. Ask it to inspect a target SKILL.md, identify important behavior, generate cases, run them, interpret failures, and improve either the skill or its tests. Subsequent failures can become regression cases rather than disappearing into a chat transcript.
This is a smart interface for a configuration-heavy tool. Authors can start from intent instead of memorizing every YAML field. The helper also checks whether the underlying CLI exists and guides installation. Codex and Claude Code users can add it globally with the skills package command, then work inside the target project.
Automation introduces a judgment risk, however. A passing suite is not automatically a good suite, especially when the same model family helps write the instruction, generate the test, and grade the result. Developers still need to review whether cases represent real requests, whether failures indicate a broken skill or a broken expectation, and whether an agent judge received enough independent evidence. Rule and script judges are preferable when correctness can be stated deterministically. Model judges belong where semantics genuinely resist exact checks.
Manual use is approachable but not trivial
The binary installer downloads a release for the current platform, while source builders need Go 1.25 or newer. A minimal suite consists of an eval.yaml file and one case file. Defaults keep the first run short: one worker, ten turns, a five-minute timeout, and JSON output. When the eval folder sits beside SKILL.md, Skill Up automatically installs the local skill for the run. A separate validate command catches configuration mistakes before model calls begin.
Real suites quickly add more decisions. Authors must provision the chosen agent CLI and credentials, create fixtures, manage temporary workspaces, separate output artifacts, and control parallelism against API limits and cost. The none environment is useful for plain-text skills, but issue #119 explains that it starts in a generated temporary directory rather than an explicitly selected existing repository. Long end-to-end cases may have to repeatedly tell the model where to change directories, a fragile workaround when repository context is the subject of the test.
CI is credible rather than decorative. The included GitHub Action runs in a Linux container with agent CLIs prebuilt, accepts model and provider settings, and can test more than one engine. Its versioning deserves close reading. The runner image pins a Skill Up binary and uses an immutable image digest, but a new CLI release does not automatically refresh that image. Older release tags may predate the action entirely. Teams wanting reproducibility should pin a known post-refresh commit or verified tag and inspect the version printed in logs.
Cross-engine support has sharp edges
Supporting several autonomous clients is Skill Up's biggest advantage and its largest maintenance burden. Release v0.9.0 fixed session lookup, cross-workspace resume protection, multi-turn response scoping, and the required CI gate. Those are encouraging fixes, but they show how much client-specific transcript and session behavior the tool must normalize.
Two open bugs are especially relevant. Issue #172 reports that Codex JSONL parsing stops silently when one record exceeds 1 MiB, potentially leaving an earlier progress message as the supposed final result. A large search output inside an agent-judge run can therefore produce a misleading grading error even when valid final JSON exists later. Pull request #184 is working on that defect, but users should still inspect raw output when Codex judging fails unexpectedly.
Issue #104 covers a different boundary: the agent judge inherits the case engine even when its configured model uses another provider's protocol. A Codex evaluation paired with an Anthropic judge can fail before meaningful grading. Use a protocol-compatible judge or test the exact engine and model pairing before standardizing a suite.
Skill routing is another unfinished area. The current evaluator measures task performance once a skill is installed. Issue #166 proposes tests for whether a skill should have triggered, should have stayed quiet, or shadowed another skill. Until that exists, teams need a separate way to test descriptions and activation behavior.
Young, active, and already useful
Skill Up was pushed on August 13, 2026, one day after v0.9.0, with active pull requests addressing parser and documentation work. The open count of 16 combines issues and PRs, so it represents a small live queue rather than 16 confirmed bugs. Releases have arrived rapidly since May, which signals energy but also a moving interface. Pinning matters.
The English README is the default, and a Chinese README and bilingual user manual are available. Documentation covers the quick start, configuration, cases, judges, reports, custom engines, CI, Windows, and Anthropic migration. Apache 2.0 licensing is friendly for company use.
For dedicated Agent Skill development, Skill Up is easier to recommend than a generic evaluation framework because it owns the awkward parts: installing the skill, invoking real coding agents, capturing their work, and comparing a baseline. Promptfoo is broader, OpenAI Evals is better aligned with model benchmarks, and OpenEvals supplies evaluator components without this execution loop. Choose Skill Up when the thing under test is specifically an Agent Skill, then keep humans responsible for the quality of the tests and the meaning of a pass.