AnyJev reached 791 GitHub stars in the six days after its repository appeared, all for asking an LLM to stop early. Many model calls should end before the model generates a single token. The Apache-2.0 project from Nokia Applied Research reads an intermediate state and returns a choice, yes-or-no answer, or score with probabilities that application code can inspect.
The timing helps explain the attention. Developers often make models produce text, then parse that text back into a route or boolean. AnyJev treats those cases as decisions from the start. Its 0.1.0 release can stop a Qwen model partway through its blocks, apply a small question-specific head, and skip generation. That could fit support routing, tool selection, escalation, and other agent steps where a paragraph is wasted work. The benchmark numbers deserve a closer look because their limits decide where the tool can be trusted.
The output layer is optional
A conventional LLM classifier asks the model to emit a label such as billing or technical. Some systems constrain the answer to one token and apply softmax to the candidate-token logits. That is quick to build, though the resulting number is easy to overread as confidence. AnyJev's method report says raw answers changed 23% of the time when the maintainers reversed the order of options in one Qwen3-8B BANKING77 test.
The project's first correction, called L0, runs cyclic shifts of the option order and adjusts for the model's label preference. It requires no labelled examples. In the published ablation, the reversal flip rate fell from 0.230 to 0.073, accuracy rose from 0.747 to 0.803, and expected calibration error fell from 0.240 to 0.184. These are the maintainers' results on 300 test items, not an independent benchmark. L0 also needs one prefill per option, so removing order bias has a compute cost.
L1 adds temperature calibration using 100 to 500 labelled examples for each question. It does not change which option ranks first. According to the documented level contract, it tries to make a reported probability correspond more closely to observed accuracy. That distinction matters when an application sends a ticket straight through at 0.95 but puts a 0.55 result in a review queue. A top choice without usable confidence still leaves the application unable to decide when the model should act.
A small head exits around two-thirds depth
Version 0.1.0 adds L2, the part of AnyJev that drew the clearest developer consequence from the research. The method document says L2 takes the hidden-state vector at the final prompt position, often after roughly two-thirds of the model's blocks. A closed-form linear head maps that vector to the available options. No gradient step changes the base model's weights.
Fitting still needs examples. A head belongs to one model and one question, and the maintainers recommend 100 to 300 labelled states in normal use. Five-fold cross-validation chooses a block, either a shrunk-LDA or ridge solver, its parameter, and a temperature. The winning head is then refit on the complete calibration set. Inference sends one prompt through the shortened model and returns the head's probability distribution.
That design cuts out the later transformer blocks, final language-model head, token sampling, and response parsing. It also changes what gets deployed. The project's Qwen3 artifacts are small JSON files containing the head, feature statistics, temperature, and question layout. The 0.1.0 changelog says a stored artifact is rejected if it is loaded against another model or incompatible question layout. Application code can require a minimum result level, preventing an L0 fallback from quietly entering a path that expects an L2 probability.
The public API makes the intended job plain:
from anyjev import Decider, Question
from anyjev.backends.vllm import VLLMBackend
d = Decider(
VLLMBackend("http://localhost:8000", "./qwen-b18"),
level="L2",
)
route = Question.choice(
"Which team should handle this?",
["billing", "technical", "sales", "other"],
name="route",
)
d.fit_head(route, states, labels, layers=[-1])
result = d.decide(ticket, [route])["route"]
The model server setup sits outside that snippet. The returned decision carries an option distribution and a declared calibration level, instead of text that happens to contain a label.
The speed claim comes with a measurable trade
On the project's typed-decision results, a Qwen3-8B L2 head reached 0.771 accuracy after 24 of 36 blocks. The reported batched cost was 0.68 times one full forward pass for both 110-token and 1,000-token inputs. Qwen3-4B reached 0.786 after 24 of 36 blocks at 0.67 to 0.69 times the full-forward cost. The heads used 300 labelled decisions per question, with 2,000 decisions held out across 20 questions.
Those figures do not say that less computation magically improves a general-purpose model. The head is trained for a fixed decision, and intermediate activations can separate its options before the remaining blocks turn that state into token probabilities. The benchmark method chooses the shallowest block whose pooled calibration accuracy comes within half a percentage point of the best tested block. Short prompts on the 1.7B model were launch-bound, producing no measured speed saving despite stopping at 18 of 28 blocks.
Calibration produces the more useful result. In the BANKING77 ablation, the share of items the project classed as automatically decidable at no more than 5% error rose from 7.7% with raw logits to 46.3% at L0 and 52.0% at L1. The maintainers warn that the 5% coverage figure has high variance with only 300 test items. Even with that caveat, it identifies the product question AnyJev is trying to answer: how much work can software route automatically at a chosen error tolerance?
Version 0.1.0 has a real serving path
AnyJev is more than a notebook attached to benchmark tables. PyPI lists version 0.1.0 as released on September 26 for Python 3.10 and newer. The package is marked pre-alpha, a fair warning for a project whose first package appeared five days earlier. Five Qwen3 artifact sets ship in the repository.
For L2 serving, the project can truncate a Hugging Face checkpoint and run it as a vLLM embedding server. The pooler returns the last-position hidden state without normalization or softmax, and the AnyJev client applies the head. Its README reports 99% identical answers between transformers and vLLM in a BANKING77-20 parity test, with a mean absolute probability difference of 0.0011. The repository also supplies a pipeline command that truncates a model, starts the server, fits a head, measures held-out accuracy and calibration, compares latency with full depth, then shuts the server down.
That measurement path deserves attention because a shorter checkpoint is not automatically faster in every serving shape. The changelog records a withdrawn prefix-caching claim that did not survive repeated tests on vLLM's embedding path. It also says FP8 improved some single-question timings while costing two accuracy points in the measured run, so the README does not recommend it. Publishing a failed performance claim and the replacement measurements gives evaluators something more useful than a best-case latency number.
L2 still needs labels
The phrase "no training" needs careful reading. AnyJev does not fine-tune the base model, and L0 needs no labels. The strongest L2 numbers still rely on a per-question head fitted with labelled examples. Change the available options or switch the base model, and the head must be solved again. The current limitations say only five Qwen3 sizes have shipped L2 heads so far, while the letter-based readout caps a choice at 26 options.
Rewording is handled separately. A head can route to another phrasing with the same options, then re-estimate its feature mean and scale from unlabelled traffic. The 0.1.0 results put a Qwen3-8B head between 0.65 and 0.70 after question rewording, down from about 0.77. Thirty unlabelled requests brought it back to 0.74 to 0.75. That is useful recovery, though it remains below a full labelled refit. A new question still needs new labels.
The main benchmark is easy to misread. Its gold answer is the mean of three samples from a teacher LLM, and a fresh sample from that teacher agrees with the gold 73.5% of the time. Accuracy there measures agreement with that teacher. It does not establish that AnyJev makes the right routing decision for a bank, support desk, or autonomous agent. Maze and Minesweeper experiments were less flattering: the project's limitations section says no tested readout beat a trivial baseline on those tasks.
What to watch next
The roadmap puts a real agent-loop evaluation near the front of the unfinished work. That test should compare generated decisions, L0, and L2 inside the same public agent benchmark while reporting task success and per-episode cost. Support for model families beyond Qwen3, a readout for more than 26 options, and a rule for abstaining at a target error rate are also pending.
The 791-star opening-week rush makes sense: AnyJev removes token generation from jobs that already have a fixed set of answers. Stars cannot settle whether the head will hold up inside a long-running agent. The next useful number is an end-to-end failure rate from the planned agent evaluation, where one wrong route changes what happens afterward. Until that arrives, AnyJev is best tested on a narrow queue with held-out labels and a human path for low-confidence results.