mrkeyoor.comAsk

logan-markewich/jefftested 06 Oct 2026commit 34b32f9

jeff review

Trial jeff for local Jev-compatible decisions, with a workload comparison as the next step.

  • Install 51 s
  • Build 1 s
22/24passed, run exited non-zero
  • 124 packages
Self-Hostedevaluationupdated 31 Oct 2026
+2stars / 7d

Trial jeff when you need a self-hosted Jev-compatible endpoint and can validate its answers before migration. jeff implements TypeSafe's Jev System One API with GLiFormer (400M parameters). API compatibility does not establish equivalent answers or production readiness: our run on October 6, 2026, at commit 34b32f9 ended with a nonzero test exit, and it did not validate model inference, deployment, accuracy or throughput.

Verdict

Trial jeff for local Jev-compatible decisions, with a workload comparison as the next step. Our recorded run established installation and build outcomes but ended at SDK import failures; it did not validate model behavior or deployment. Keep hosted Jev in consideration for reasoning-heavy tasks, use SetFit when training a classifier is the goal, and consider BentoML when serving infrastructure is the missing piece.

We ran it

Lab card: what happened when we ran jeff
Lab card from our sandbox run
Screenshot of jeff (github.com/logan-markewich/jeff)
github.com/logan-markewich/jeff
Install✓ · 51s124 packages · 5774 MB
Build✓ · 1s
Tests✗ · 9s22 passed · 2 failed · 6 skipped of 24 (pytest)
Known vulns0(pip-audit)
Repo68 files~4,564 lines of source · 10.5 MB · 0 CI workflows · tests dir

Answers from our run

Does jeff build from source?

Dependencies installed in 51 seconds (124 packages), and the build succeeded in 1 seconds. We cloned commit 34b32f9 into a clean Debian container with 3 CPUs and no project-specific setup.

Do jeff's tests pass?

No. The project's own test command finished in 9 seconds and reported 22 passed, 2 failed, 6 skipped (pytest).

Does jeff have known vulnerabilities in its dependencies?

pip-audit found none in the dependency tree at the time of our run.

Who should not use jeff?

Teams requiring a clean test result from the recorded environment.

What are the alternatives to jeff?

TypeSafe Jev, SetFit, BentoML. Trial jeff for local Jev-compatible decisions, with a workload comparison as the next step.

Setup2/5Under the visible rubric, completed installation is offset by SDK import failures and the recorded 5,774 MB installed environment.
Docs4/5Under the visible rubric, detailed setup, configuration and API guidance includes explicit behavioral and benchmark limitations.
Community2/5Under the visible rubric, reviewed support evidence is thin: a concrete issue report does not establish response quality or adoption.
Maturity2/5Under the visible rubric, the nonzero test exit and absence of detected CI and container packaging limit confidence at the recorded revision.

Who it’s for

Teams with existing TypeSafe SDK clients that want to evaluate local hosting.
Python operators prepared to download model weights and manage the inference service.
Applications asking bounded choice, ordered-score or yes-probability questions.
Engineers who can compare production questions, batching and thresholds before migration.

Who it’s NOT for

Teams requiring a clean test result from the recorded environment.
Operators treating SDK compatibility as proof of identical model behavior.
Applications requiring question independence without evaluating the isolation setting.
Buyers choosing on promised savings or runtime performance: our sandbox measured neither.

Setup reality

Our run on October 6, 2026, used commit 34b32f9 in our sandbox. The attached harness recorded successful install and build outcomes, then a test exit code of 1. Install took 51 seconds and build took 1 second; the test step took 9 seconds. The installed environment contained 124 packages and occupied 5,774 MB.

The stored parser summaries report 22 passed, 2 failed, total 24, and 6 skipped. These fields do not establish an exhaustive count of unique tests, and the skips must not be folded into a claim that only 24 tests ran. The raw footer separately reports 2 failed, 22 passed and 6 skipped. Both named failures stopped at an import of typesafe_sdk.

The documented setup requires uv, Python 3.12 and downloaded model weights. The recorded build success does not establish that a model-backed server or deployment worked. Pip-audit recorded zero advisories, but scan coverage is unknown because scannedPackages is absent.

Should you replace hosted Jev with jeff?

Evaluate jeff if control over hosting is worth operating and validating a local classifier. The README describes compatibility with the official TypeSafe SDK through TYPESAFE_BASE_URL and supports choice, score and noul questions. That gives an existing client a migration path, but it does not settle whether its decisions remain acceptable.

The project README says jeff is less accurate than Jev on reasoning-heavy tasks. Keep hosted Jev in the comparison when that accuracy matters. This is the project's comparison, not a result from our sandbox, and we did not verify its cost estimates.

What our sandbox established

Our run on October 6, 2026, used commit 34b32f9 in our sandbox. It used the recorded Python 3.12 Bookworm image with 3 CPUs and 8 GB of memory. Installation succeeded in 51 seconds, with 124 installed packages occupying 5,774 MB. The build step returned zero in 1 second; the test step returned 1 after 9 seconds.

Our harness stores command outcomes as success flags, exit codes, elapsed times and captured output tails. It stores parsed test counts separately. The testing-methodology page explains those categories and their limits; its current procedure is not evidence that this older run used today's methodology version.

The attached stored parser summaries contain 22 passed, 2 failed, total 24, and 6 skipped. We preserve those values without treating them as an exhaustive unique-test inventory. The raw pytest footer separately says "2 failed, 22 passed, 6 skipped, 1 warning in 5.29s". Its duration describes the pytest footer, while 9 seconds is the stored harness test-step duration.

The two named failures were tests/test_sdk_live.py::test_sdk_roundtrip and tests/test_sdk_live.py::test_sdk_errors. Both stopped with ModuleNotFoundError: No module named 'typesafe_sdk'. The README says the development sync installs that SDK and that these tests use a live server with a fake backend. The captured failures establish an import problem in this environment; they do not establish a model defect or explain why the dependency was missing.

We did not download model weights or run model inference, the benchmark suite, GPU deployment or a production workload. The README says model integration tests skip when the required local weights are absent. Skipped tests are not failures, and a successful build command does not fill those coverage gaps.

Check question independence before migration

The README says choice and score questions share an encoder pass and can affect each other. It documents JEFF_ISOLATE=all for separate passes at extra cost. Issue 4 reports a choice changing after the question order was reversed at the same commit as our run. That is another user's report, not an observation from our sandbox.

The README also explains that score uses the raw distribution, while displayed probabilities are temperature-scaled. It says the weighted average matches only at JEFF_TEMPERATURE=1. A caller calculating a score from returned probabilities therefore needs to check that configuration rather than assume the values are interchangeable.

Before migration, replay representative requests in their intended batches and order. Compare labels, probabilities and threshold decisions with the existing service. Repeat the comparison with isolation enabled if independent questions are a requirement. This is a proposed validation step, not work completed in our run.

Account for operation and evidence gaps

The documented device preference is CUDA, then MPS, then CPU. The README describes CPU as a fallback and provides an ONNX option that leaves the recurrent and classification layers in PyTorch. Hardware support in documentation is not a compatibility test from our Debian sandbox.

Authentication defaults to off when JEFF_API_KEYS is empty. Configure bearer keys before exposing the service. The recorded checkout had no detected CI workflow, Dockerfile or Compose file; that observation applies to the recorded revision, not every future version.

Pip-audit recorded zero advisories. The supplied record has no scannedPackages field, so scan coverage is unknown. This result is neither a security review nor proof that all installed packages were checked.

Which alternatives answer a different need?

Hosted TypeSafe Jev is the direct service to compare against because jeff implements its request contract. The supplied README supports an accuracy comparison, but we did not independently test either service's accuracy.

SetFit is a framework for few-shot fine-tuning of Sentence Transformers. Consider it when you have labeled examples and want to train a task-specific classifier. Its supplied documentation establishes a training workflow, not Jev API compatibility.

BentoML is a Python library for building online serving systems for model inference. Consider it when the job is packaging and serving your own model API. It is serving infrastructure and can complement a classifier; it does not by itself replace the classifier's decisions or establish Jev compatibility.

How to read the ratings

These are editorial assessments, not benchmark results. The rubric is 1 for major unresolved obstacles, 2 for demonstrated obstacles or thin supporting evidence, 3 for an adequate documented path, 4 for detailed guidance supported by the reviewed material, and 5 for broad validation beyond documentation. An unassessed dimension would receive no numerical rating.

Setup ease is 2 because installation completed but the SDK tests failed and the installed footprint was substantial. Documentation is 4 because the README explains setup, configuration, API behavior and benchmark limitations. Community is 2 for the narrow support evidence reviewed here: a concrete issue report, without established response quality or adoption evidence. Maturity is 2 because the recorded test command failed and the checkout had no detected CI or container packaging. These ratings do not measure popularity, security or production reliability.

Alternatives

ProjectWhat it isPick it when
TypeSafe JevThe service whose System One API jeff implements.Keep it in consideration when reasoning-heavy accuracy matters; jeff's README reports an accuracy disadvantage, which our sandbox did not independently test.
SetFitA framework for few-shot fine-tuning of Sentence Transformers.Choose this route when labeled examples support training a task-specific classifier; the supplied source does not establish Jev API compatibility.
BentoMLInfrastructure for building and serving model inference APIs.Consider it when you need to package and serve a custom model. It complements a classifier rather than supplying equivalent Jev decisions.

What people are saying

  1. [hackernews] Jeff – Jev-compatible 0.8B decision models, trained at home, ~30 ms
  2. [velocity-scout] firelex/jeff
  3. [velocity-scout] logan-markewich/jeff

Sources

  1. jeff README
  2. jeff repository
  3. Question-order issue
  4. Open Docker support pull request

More self-hosted reviews