Can skill-up evaluate Codex skills?
The README lists Codex among the supported engines and describes comparisons with and without a Skill. That makes skill-up a candidate for regression testing Codex skills, but our sandbox run did not validate that integration. Start with one behavior whose expected response or file change you can check directly before relying on model-generated grades.
skill-up evaluates Agent Skills, agents, and workspaces. Its declarative cases configure the environment, engine, inputs, and judging method. The documented runner prepares a workspace, optionally installs a Skill, invokes an engine, grades the response and workspace changes, and writes reports. Rule, script, and agent judges serve different needs; a model judge adds another component whose behavior you must evaluate.
What did our sandbox establish?
Our run on October 1, 2026, at commit 7f1ff9b recorded a successful npm install in 17 seconds. This is an existing stored run, not a new execution. The full recorded commit is 7f1ff9b8e2d7c654728de526867f2f7e7b78ea51.
The harness recorded install success, exit code 0, elapsed time, and no timeout. It recorded build and test as skipped, each with the reason "no script". Its detector found the documentation scripts docs:dev, docs:build, and docs:preview, so those skips do not mean the repository lacks build machinery. They mean this npm-oriented run did not invoke that machinery or test the Go CLI.
The stored parser summary records 150 dependency packages occupying 95 MB. The captured install tail instead says 127 packages were added and 128 audited. These values describe different recorded outputs; we have not reconciled them or established exhaustive package coverage.
The audit records also disagree: the stored summary reports zero advisories, while the install tail reports four vulnerabilities, including two high and two moderate. These are recorded advisory results, not findings from a security review. Scan coverage is unknown. This evidence does not support describing the installation as vulnerability-free.
The MrKeyoor testing methodology explains how command outcomes and measurements reach reviews. It provides provenance context; its current procedure does not establish which methodology version this older run used. The attached run record determines the scope described here.
What can you evaluate beyond a Skill?
The commit-matched README documents agent-only evaluations and cases that check repository work. For an agent-only evaluation, it says to set skills: [] explicitly because omitting skills can cause a discovered SKILL.md to be installed automatically. That distinction matters when your baseline is supposed to exclude a Skill.
The same README documents evaluation against an existing local workspace. That mode requires the host runtime, one case at a time, and benchmark mode disabled. Although the directory is preserved, agent actions and case setup can change its contents. Use a disposable copy when those changes need inspection before acceptance.
Structured output includes result.json, Anthropic-compatible grading and benchmark reports, JUnit XML, and HTML. These formats help move results into review and CI; they do not establish that the chosen cases cover the behaviors you care about.
Can the judge use a different engine?
The current README documents a separately configured judge.engine that does not inherit the evaluated agent's model, CLI overrides, or connection configuration. It distinguishes that setting from legacy judge.model, which reuses the evaluated agent's engine. Prepare the judge's engine and credentials separately.
This current documentation replaces the old page's blanket warning about cross-provider judging. It does not prove that independent judging worked at our measured commit: we did not run a judge, and the commit-matched README does not contain the current role-configuration section. Validate your exact agent and judge pairing before using its grades to accept changes.
What setup remains outside our run?
The documented CLI workflow needs evaluation cases and a configured agent engine; model-backed execution also needs the relevant access and credentials. The bundled skill-upper can guide installation and evaluation, but we did not test that conversational workflow.
The supplied Windows guide, pinned to commit 8566175, recommends WSL2 for full agent evaluations. It documents native Windows support for host execution and script judges, while explaining that the agent Node/nvm bootstrap is Bash-based. That is documentation for the pinned revision, not a Windows result from our sandbox.
The README also documents a Linux-only Docker GitHub Action with a CLI version pinned inside its runner image. Publishing a CLI release does not automatically refresh that image. Check the effective CLI version in the Action log rather than assume the image matches a release tag. The supplied v0.9.1 release notes describe fixes, including Codex JSONL handling; they are not evidence that our later npm run exercised those fixes.
Which alternatives answer a different need?
Promptfoo is a CLI and library for evaluating and red-teaming LLM-based apps. Consider it when the evaluation target is an application, prompt, or model comparison rather than primarily a Skill and its workspace behavior. This is a difference in documented scope, not a measured performance comparison.
Anthropic Skills provides examples of Claude skills for inspiration and authoring reference. It complements an evaluator by supplying material to study or adapt; it does not replace the evaluation runner described here. Its README advises testing the skills in your own environment before relying on them for critical tasks.
How to read the editorial ratings
The ratings use a five-point editorial rubric: 1 means substantial documented obstacles or little supporting evidence; 2 means a narrow usable path with major gaps; 3 means a usable documented workflow with material setup or validation gaps; 4 means broad documented support or visible participation with remaining limits; 5 means unusually complete evidence for that dimension. These are judgments, not benchmark results or adoption measurements.
Setup ease is 3 because npm installation succeeded but evaluator setup remains untested and requires separate agent configuration. Documentation is 5 for the supplied coverage of evaluation modes, configuration, reports, CI, and Windows limitations; this rates explanatory coverage, not the execution accuracy of every instruction. Community is 4 for visible contributions and maintenance work in the supplied repository and release material, not for an unverified current user count. Maturity is 3 because the project documents a usable evaluation workflow, while our run leaves its core execution unverified and the Action image has a separate version lifecycle.