AgentJev chooses from actions you describe
Hand AgentJev a failed-build transcript and a list of possible next steps, and it returns a probability distribution over those options. The Qwen-based model scores them without generating an answer sentence, so callers don't have to extract a decision from prose. That fits an existing agent loop with constrained actions.
Our checkout had 90 files and about 6,760 source lines, occupying 5.9 MB. Installation expanded the environment to 5,638 MB. This is a local learned model with a serving stack. If your routing rule is already expressible as ordinary code, adding that runtime needs a reason beyond preferring a model to an explicit condition.
The probability ranks supplied options, not future success
The AgentJev README separates boolean questions, choices, and ordered scores. A choice returns a distribution and the gap between the leading options. It also warns that a choice probability is relative to the candidate set. Calling the highest-ranked action safe or likely to succeed would require evidence from the job you actually run.
Our 19 passing cases don't establish decision accuracy in your incident queue or coding workflow. The repository publishes its own benchmark comparisons, but those measure agreement with a teacher's decisions rather than completed agent tasks. We didn't reproduce those comparisons in this sandbox. I'd use the distributions to prioritize low-consequence work first, record the outcomes, and decide whether the scores deserve any authority in that particular workflow.
What happened when we ran it
Our sandbox installed AgentJev commit 1c2c1b1 in 58 seconds and built it in 1 second on October 8, 2026. Installation added 74 packages and occupied 5,638 MB. The fresh Debian container had Python 3.12, 3 CPUs, and 8 GB of RAM, with no secrets or elevated privileges. The install and build both succeeded. The result does not establish usable model inference.
Pytest exited 1 after 2 seconds. The supplied summary records 19 passed, 0 failed, and 2 collection/setup errors across 21 cases. Both errors name functions in run_practical_test.py, and the shown traceback says fixture jev was not found. That is a setup failure for those functions, rather than a failed assertion about a model answer. The tail also records 6 passing subtests. Pip-audit reported 9 known vulnerabilities. The supplied report gives no affected package names, advisory details, or severity, so we cannot say which runtime paths they involve. The scan found 1 CI workflow file, no Dockerfile, and a tests directory. For a new deployment, obtain the audit details and resolve the dependency findings before putting the service near sensitive state.
Pinned weights and a local tokenizer are part of setup
The quick start retrieves weights from a pinned Hugging Face revision, verifies the release files, and wraps them into the checkpoint format the server expects. It also fetches the Qwen base model locally. The tokenizer uses local files, so a repository identifier works only when its assets are cached. A regular causal-language-model loader won't load this candidate-scoring model correctly.
The 5,638 MB we measured belongs to the installed environment, not a documented total for every downloaded weight and cache. The README recommends selecting a PyTorch build for your hardware before the requirements, and the server defaults to a CUDA device. Our CPU sandbox supplies no evidence about GPU performance or CPU serving quality. Local inference still requires matching assets and device settings.
The Claude Code hook lets tools proceed on errors
agentjev_hook.py covers Claude Code's Bash, Write, and Edit tools. The hook source confirms that an exception ends with a successful exit, allowing the tool to proceed. For file operations, it sends the path as the action summary rather than the proposed file contents. A service outage or a risky edit hidden behind an ordinary filename therefore sits outside what this hook can reliably judge.
With 2 setup errors in our full test command, we have no basis for recommending this commit as a mandatory safety control. The hook can provide another opinion for a supervised developer, provided the developer understands what it sees. Teams that require operations to stop when inspection is unavailable should walk away from the supplied hook. Enforced filesystem permissions and review requirements need to remain independent of a learned risk score.
Serving behavior depends on the checkpoint you load
The main README says the pinned release refuses overlong inputs. The service notes qualify that: training checkpoints carrying their own configuration can truncate inputs and report that count. Replacing weights can therefore change how much of a trace the scorer reads. The main quick start binds the service to loopback, and the bundled client calls its local HTTP endpoint. Publishing it as a shared service requires additional operating decisions.
The 58-second install won't settle those decisions. Choose a known checkpoint and inspect its input policy before passing long diffs or incident threads. For simpler semantic routing, compare Semantic Router. If the problem is managing state, retries, and human approvals across a workflow, LangGraph addresses that orchestration job. Neither is a drop-in copy of AgentJev's scorer, but either can eliminate the need for it when the surrounding requirement is different.
October fixes show activity, not proof of a finished service
The last push was October 1, 2026, and v1.0.0 shipped that day. GitHub listed no open issues or PRs, while recent merged changes pinned weight downloads and removed assumptions about the author's local checkout. Those concrete fixes show maintainers addressing portability. An empty queue does not explain our independent setup errors.
AgentJev deserves an experiment when you already have candidate actions, local model capacity, and a way to compare decisions with real outcomes. Its 74-package environment and 9 known vulnerabilities make adoption more involved than copying a helper function. Before wiring the scorer to a consequential tool, demonstrate that the workflow behaves correctly when the model is wrong, the checkpoint truncates state, or the server disappears.