mrkeyoor.comAsk

coldteadotai/abidetested 06 Oct 2026commit 0fc600f

abide review

Trial Abide for instructions that need judgment, with human review of its rubric and feedback.

  • Install 17 s
  • Build 6 s
6/6passed, run exited non-zero
  • 229 packages
Dev Toolsevaluationupdated 31 Oct 2026
+2stars / 7d

Try Abide as an advisory check for repository instructions that require judgment, with someone responsible for reviewing its feedback. Abide lets edits continue when credentials or the network are unavailable. It compiles instruction files into a rubric and asks TypeSafe's Jev model to judge changes made by supported coding agents. Our run on October 6, 2026, at commit 0fc600f installed and built successfully, but the test command exited 1. We did not verify a live model-driven repair session.

Verdict

Trial Abide for instructions that need judgment, with human review of its rubric and feedback. Our run installed in 17 seconds. Build completed separately in 6 seconds. The test command exited 1, and the live Pi repair test was skipped. Version 0.0.9 therefore needs live workflow validation before wider use. Its documented behavior allows unchecked edits to continue, so retain mandatory controls alongside it.

We ran it

Lab card: what happened when we ran abide
Lab card from our sandbox run
Screenshot of abide (github.com/coldteadotai/abide)
github.com/coldteadotai/abide
Install✓ · 17s229 packages · 318 MB
Build✓ · 6s
Tests✗ · 44s6 passed · 0 failed of 6 (vitest)
Repo192 files~15,270 lines of source · 10.7 MB · 0 CI workflows

Answers from our run

Does abide build from source?

Dependencies installed in 17 seconds (229 packages), and the build succeeded in 6 seconds. We cloned commit 0fc600f into a clean Debian container with 3 CPUs and no project-specific setup.

Do abide's tests pass?

No. The project's own test command finished in 44 seconds and reported 6 passed, 0 failed (vitest).

Who should not use abide?

Workflows that require unavailable checks to block an edit.

What are the alternatives to abide?

Semgrep, ast-grep. Trial Abide for instructions that need judgment, with human review of its rubric and feedback.

Setup3/5Under the visible rubric, installation and build succeeded, while credentials, agent configuration, and rubric review remain necessary work.
Docs5/5Under the visible rubric, the supplied documentation explicitly covers setup, commands, data routing, failure behavior, and live-test limits.
Community3/5Under the visible rubric, public material and review activity provide limited evidence; contributor breadth, response times, and adoption remain unverified.
Maturity2/5Under the visible rubric, our October 6 run at commit 0fc600f had a failed test command, and live repair was not verified.

Who it’s for

Developers using Claude Code, Codex, OpenCode, or Pi who want feedback on instructions that require judgment.
Maintainers willing to inspect the compiled rubric and review missed or noisy checks.
Teams adding advisory feedback alongside tests, static analysis, and mandatory review.

Who it’s NOT for

Workflows that require unavailable checks to block an edit.
Repositories whose changed code cannot be sent to the configured decision service.
Teams seeking verified live repair success or security assurance from this sandbox run.
Rules that depend on context absent from the rule and diff.

Setup reality

Our run took place on October 6, 2026, at commit 0fc600f in our lab-node:22 sandbox with 3 CPUs and 8 GB RAM. Installation completed in 17 seconds. Build completed separately in 6 seconds. The test command exited 1 after 44 seconds without timing out.

The stored dependency measurement is 229 packages occupying 318 MB. The installation log's Packages: +228 line describes a different recorded quantity and does not replace that measurement. Pnpm also reported ignored build scripts for @google/genai, esbuild, and protobufjs.

The attached stored parser summaries record 6 passed, 0 failed, 0 skipped, and 6 total. These are not verified exhaustive counts of unique tests. The raw log separately identifies a failed Picker assertion and a skipped live Pi test; neither changes the stored fields.

Documented setup requires a TypeSafe or Vercel AI Gateway key, installation into the selected agent, and compilation of instructions into .abide/rubric.json. Successful installation and compilation do not establish that authenticated checks or live repairs worked.

Should you use Abide?

Use Abide for advisory feedback on repository rules that need judgment, provided someone will review its decisions. Abide lets edits continue when credentials or the network are unavailable. That behavior makes it unsuitable as the sole control for a requirement that must block changes.

Our run on October 6, 2026, at commit 0fc600f installed and built successfully, but the test command exited 1. For version 0.0.9, the evidence supports a limited trial alongside existing checks. It does not establish that a working agent session received and completed a repair.

What our sandbox established

Installation completed in 17 seconds, and build completed separately in 6 seconds. Our sandbox used lab-node:22 with 3 CPUs and 8 GB RAM. The harness records command success, exit code, elapsed time, timeout status, and a captured log tail. Parsed test counts are stored separately from command outcomes.

The test command exited 1 after 44 seconds without timing out. The attached stored parser summaries record 6 passed, 0 failed, 0 skipped, and 6 total. Those fields do not establish exhaustive unique test coverage. No new execution or complete-log reconciliation was performed for this correction.

The raw log identifies the failure in test/ui.test.tsx, in the Picker case named "moves with the arrows and keeps only the answer on screen after enter". At line 166, the assertion expected the final frame to contain "Vercel AI Gateway key" and received a newline. This establishes the failed assertion, but not its underlying cause or whether credential selection fails in an actual session. We have not substituted package-footer counts for the attached parser summaries or combined them into a new total.

The raw log marks test/piLifecycle.live.test.ts as skipped. The Pi extension guide explains that the normal suite skips this live repair test; opting in uses configured credentials and makes paid API calls. Skipped means untested here, not failed. Our run did not verify that live repair path, a deployment into a working agent session, model accuracy, or runtime latency.

The audit field is null. Advisory results and scan coverage are unknown. The repository scan recorded no CI workflow files and no Dockerfile, which describes repository shape rather than operational reliability.

MrKeyoor's testing methodology explains the distinction between command outcomes, parsed counts, and sandbox limits. It supplies provenance context; its current text does not prove that this older run used today's procedure unchanged.

What Abide checks

Abide compiles repository instructions into .abide/rubric.json and checks edits or completed activity diffs against that rubric. The README at the tested commit documents integrations for Claude Code, Codex, OpenCode, and Pi. These are documented capabilities, not four integrations independently exercised by our run.

A score of 0.8 or above requests repair. Scores from 0.5 to 0.8 produce a note, while lower scores remain quiet. These thresholds control how a judgment is delivered; they do not guarantee that the judgment is correct. The rubric has no built-in policy rules, so useful instruction files and review of their compiled questions are part of adoption.

Jev receives the rule and diff rather than the conversation. Choose initial rules whose violations a reviewer can recognize from the change itself. Treat instructions that depend on unseen repository context with more caution.

What happens when a check cannot run?

The tested README says every hook path exits 0. Missing credentials or network access allow the edit through unchecked, with the miss logged in .abide/events.jsonl. The Pi guide also states that checks happen after mutations and are not a pre-write security boundary.

Direct checks send changed lines to TypeSafe under your key. The README documents gateway routing and a configurable direct endpoint separately. Review the provider and retention arrangement that applies to your configuration before enabling checks on private code. Our sandbox did not validate those arrangements.

Does the replay prove agents repair violations?

The author's September 18, 2026 replay did not measure whether agents repaired flagged changes. It reconstructed edits from recorded Claude Code sessions rather than running those sessions again. Its reported reviewer was Claude reading each rule strictly, rather than an independent human audit.

The replay report describes 93 sessions, 1,256 judged edits, and 147 judged turns. Claude confirmed 10 of 39 edit flags and 11 of 15 turn flags, producing the report's precision figures of 26% and 73%. These are upstream replay results, not passing-test counts or measurements from our sandbox.

The sample used repository-specific rubrics, excluded shell-command changes, and did not establish accuracy on another repository. Its introduction and method also differ over the repositories' public or private status. The useful next step is to inspect your own rubric and feedback, rather than assume the reported precision transfers to your rules.

When should you choose Semgrep or ast-grep?

Semgrep is a static analysis tool that can run as a pre-commit check or in CI/CD workflows. Choose it when a requirement can be expressed in its analysis system and belongs at those workflow stages. Its README warns that Community Edition analysis is bounded by a function or file. It can replace an individual Abide check that it can express, while complementing Abide for other requirements.

ast-grep searches, lints, and rewrites code by matching syntax-tree structure. Choose it for a recognizable code pattern or structural rewrite. Its README establishes that role. It can replace model judgment for an expressible structural rule or run alongside Abide when other instructions still require judgment.

How to read the ratings

These are editorial ratings within each dimension: 1 means substantial obstacles, 2 means material limitations, 3 means usable with ongoing work or limited evidence, 4 means strong evidence with minor gaps, and 5 means unusually complete supporting evidence. They are not measured product performance or adoption scores.

Setup ease is 3: installation and build succeeded, but credentials, agent configuration, and rubric review remain. Documentation is 5 for its explicit treatment of setup, commands, data routing, failure behavior, and live-test limits. Community is 3 on limited evidence of public material and review activity; contributor breadth, response times, and adoption remain unverified. Maturity is 2 because our test command failed and live repair was not verified.

What to try first

Pin version 0.0.9, inspect the generated integration and rubric, and start with rules whose violations are easy to confirm. Review repair requests and logged unchecked edits before expanding use. Keep mandatory tests and static analysis in place. The next evidence to collect is whether your chosen agent completes a useful repair for your own rules.

Alternatives

ProjectWhat it isPick it when
SemgrepStatic analysis for finding code issues and enforcing coding standards, with pre-commit and CI/CD integration.Choose it when a requirement can be expressed in its analysis system. It can replace that individual Abide check or complement checks that still need judgment. Account for Community Edition's function and file boundaries.
ast-grep gh↗A CLI for structural code search, linting, and rewriting through syntax-tree patterns.Choose it when a rule or rewrite depends on recognizable syntax structure. Use it for that pattern and retain Abide only where model judgment adds useful feedback.

What people are saying

  1. [velocity-scout] coldteadotai/abide

Sources

  1. Abide README
  2. Abide replay benchmark
  3. Pi extension limits and live verification
  4. Repository-context issue 24
  5. Quoted-filename fix pull request 34
  6. GitHub repository metadata

More dev tools reviews