mrkeyoor.comAsk

SemiAnalysisAI/InferenceXtested 04 Nov 2026commit 3b4b94f

InferenceX review

Our InferenceX sandbox built in 4 seconds, but tests exited 1 with 47 failures and 153 collection/setup errors.

  • Safety failed
  • Install 10 s
2,767/2,967passed, run exited non-zero
  • Build 4 s
  • 41 packages
AI Toolsevaluationupdated 17 Nov 2026
+26stars / 7d

InferenceX is SemiAnalysis's open-source platform for comparing how hardware and serving software run large language models. It publishes benchmark recipes and results, including AgentX sessions that exercise growing conversations and reusable context.

Verdict

Our InferenceX sandbox built in 4 seconds, but tests exited 1 with 47 failures and 153 collection/setup errors. Use it if you can operate inference infrastructure and reproduce a workload-specific chart point. Skip it if you need a clean local test gate or expect published recipes to settle deployment parity by themselves.

InferenceX infographic: compare hardware and serving configurations, plus our test result and who it suits
Tested 4 November 2026: 2,767 of 2,967 source tests passed, 47 failed. Before that: source install 10 s, build 4 s.

We ran it

Lab card: what happened when we ran InferenceX
Lab card from our sandbox run
Screenshot of InferenceX (inferencex.com)
inferencex.com
Install✓ · 10s41 packages · 48 MB
Build✓ · 4s
Tests✗ · 232s2767 passed · 47 failed · 153 errors of 2967 (pytest)
Known vulns0(pip-audit)
Repo1801 files~285,191 lines of source · 32.4 MB · 29 CI workflows

Answers from our run

Does InferenceX build from source?

Dependencies installed in 10 seconds (41 packages), and the build succeeded in 4 seconds. We cloned commit 3b4b94f into a clean Debian container with 3 CPUs and no project-specific setup.

Do InferenceX's tests pass?

No. The project's own test command finished in 232 seconds and reported 2767 passed, 47 failed, 153 collection errors (pytest).

Does InferenceX have known vulnerabilities in its dependencies?

pip-audit found none in the dependency tree at the time of our run.

Who should not use InferenceX?

Buyers expecting one curve to describe a single deployable configuration: AgentX points can use different optimizations.

What are the alternatives to InferenceX?

AIPerf, metrumbench-llm within Metrum AI Bench. Our InferenceX sandbox built in 4 seconds, but tests exited 1 with 47 failures and 153 collection/setup errors.

Setup4/5Our source checkout installed in 10 s and built in 4 s with 41 dependencies, but we did not run a README task.Editor’s note: Supported accelerator setup was not reproduced; local source checks passed install and build but failed tests, and the Compose-based attempt stopped before deployment.
Docs2/5The 6,590-character README has no install instructions or usage instructions or examples.Editor’s note: We read execution and provenance guidance, but did not assess successful use of those instructions across the supported hardware matrix.
Community4/5Last push on 2026-11-17, 2,050 stars, maintainers answered or closed 10 of the 10 most-discussed issues we sampled and 171 open issues and pull requests is a heavy backlog for its size.Editor’s note: Selected issue threads show engineering work and unresolved questions; they do not support a numerical judgment of support capacity or adoption.
Maturity2/5Latest release h3-v2-release-v1-references on 2026-11-07, repository created 2025-07-12 and its test suite did not pass in our sandbox (2,767 of 2,967 passed).Editor’s note: GPU execution, repeatability and production reliability were not assessed by our source sandbox or the blocked hands-on attempt.

How we score

Who it’s for

Inference engineers comparing specific hardware, serving engines and workload configurations.
Operators with an OpenAI-compatible endpoint who want to replay AgentX sessions.
Researchers prepared to archive artifacts and reproduce a selected chart point.

Who it’s NOT for

Buyers expecting one curve to describe a single deployable configuration: AgentX points can use different optimizations.
Contributors requiring a passing fresh-sandbox test gate: our command exited 1 with 47 failures and 153 collection/setup errors.
Teams requiring permanent access to every historical artifact: the About page states 90-day GitHub retention, and issue #1691 reports expired logs.

Setup reality

Our November 4, 2026 sandbox at 3b4b94f installed in 10 seconds with 41 dependencies and 48 MB on disk, then built in 4 seconds. Tests exited 1 after 232 seconds: 2,767 passed, 47 failed and 153 collection/setup errors out of 2,967. pip-audit reported 0 known vulnerabilities. The tail repeatedly shows a missing marshmallow import in launch snapshot tests.

The local Python project lives in inferencex-e2e/. Running AgentX also needs a reachable OpenAI-compatible model server and resolvable tokenizer, with credentials when those resources require them. Hosting the separate dashboard adds PostgreSQL. API skills read existing results without launching benchmarks.

The sandbox had 3 CPUs and 8 GB RAM; its image label does not establish actual runtime versions. A November 13 hands-on attempt stopped because our runner required a Compose file that was absent. Teardown succeeded. The generic summary's deployment wording conflicts with the failed step, so it supplies no successful deployment evidence.

Short answer: use the charts to choose a recipe to reproduce

Our InferenceX sandbox built in 4 seconds, then its test command failed with 47 failures and 153 collection/setup errors. Use the project if you run inference infrastructure and can investigate the exact configuration behind a promising result. Skip it if you need a ready-made hardware buying recommendation: changing the workload, routing or GPU denominator changes the comparison you are making.

InferenceX is SemiAnalysis's open-source inference benchmarking platform, formerly called InferenceMAX. Its useful output is a comparison tied to a model, serving engine and workload. Start with one point that meets your response-speed target, save the configuration and artifacts, and reproduce that point on your own deployment. The public records show why reproduction matters: a published recipe can leave important deployment questions unanswered. Platform description, reproduction report.

The benchmark, dashboard and client have separate jobs

The InferenceX repository runs benchmarks, the dashboard displays results, and API skills query existing observations. The AgentX client replays traffic against a model server. These components answer different questions and have different setup requirements. Reading public results starts with the dashboard; replaying a workload starts with a reachable serving endpoint; operating your own dashboard adds its database. Choose the component that matches the work you intend to do.

InferenceX compares fixed-sequence serving and AgentX, a long-context workload that replays dependent coding-agent sessions. The distinction changes what a result means. Independent requests expose baseline chip and kernel performance. AgentX brings shared prefixes, growing conversations, pauses and subagents into the measurement, so routing and cache behavior affect the outcome. Pick the scenario that resembles the traffic you intend to serve. Workload explanation.

Component Job What you provide
SemiAnalysisAI/InferenceX Benchmark recipes and execution tooling Serving infrastructure for the selected recipe
SemiAnalysisAI/InferenceX-app Dashboard and database-backed visualization PostgreSQL when hosting your own copy
AgentX harness / AIPerf scenario Replay traffic against a model server Reachable OpenAI-compatible endpoint and tokenizer
InferenceX API skills Query existing observations Public HTTPS access; queries do not launch benchmarks

The official benchmark README links a standalone AgentX route that avoids CI and Slurm. The dashboard README and API examples describe the other two paths. Start with the public dashboard if your question is about existing results. Host the application only when you need to operate that application yourself.

GitHub's public API returned 2,050 stars, 171 open issues and PRs, and a last push on 17 November 2026. The sampled tracker includes both detailed investigations and an unresolved reproduction request closed for inactivity. That combination supports calling the repository active; it does not establish a support commitment or widespread successful reproduction. The latest-release endpoint returned h3-v2-release-v1-references, published on 7 November. Read that reference tag alongside current commits rather than treating a release tag as a complete account of project health. Public repository metadata.

What people use InferenceX for

The issue records show engineers using InferenceX for reproduction attempts and instrumented serving experiments, with outcomes that need reading beyond the issue's closed label.

  • GitHub user OrZipori attempted a DeepSeek Dynamo TRT-LLM plus MTP reproduction on four B200 SXM nodes using Kubernetes. OrZipori writes, "Current best result is ~8.3K per decode gpu." The referenced published result was 10,012 tokens/s/GPU. Maintainers suggested trying Slurm first; the thread closed for inactivity without a successful reproduction. OrZipori reports throughput per decode GPU, while the published result is labeled per GPU; the thread also contrasts Kubernetes with Slurm. Those differences prevent treating the gap as a matched regression. Issue #1691.
  • functionstackx asked the repository's Claude automation to sweep gpt-oss-120b FP4 on H100 with power collection. The automation reported 22 power CSVs plus benchmark results. When its first analysis focused on TP8, functionstackx requested coverage of all tensor-parallel configurations. That correction is a useful reminder to inspect what an automated analysis actually included. Issue #828.
  • functionstackx also requested DeepSeek SGLang sweeps with GPU monitoring. An early fallback run omitted the requested telemetry; later branch-specific B200 work reported five successful configurations and analysis of benchmark and GPU power results. A completed serving run and a completed instrumentation task are separate outcomes. Issue #870.

Outside the tracker, Together researcher simran_s_arora claims its compiler beat the best InferenceX engine across the settings it tested. BackboardIO also posts comparisons against selected InferenceX baselines. These are vendor-authored comparisons with selected workloads, rather than our measurements or independent confirmation. They show how the benchmark is used as a reference point, not how many production deployments depend on it. Collected X posts.

What happened when we ran it

Our November 13 hands-on attempt stopped at a safety check because the repository had none of the Compose filenames required by that runner. The record calls it an "unsafe repo," but the reason shown is the runner's Compose requirement. That label is no finding about malicious code. Teardown reported success. A generic run summary says it deployed with Docker Compose and loopback ports; the failed step contradicts that summary, so we count this as a blocked attempt.

Our separate source sandbox ran commit 3b4b94f on November 4, 2026, with 3 CPUs and 8 GB of RAM. The Python project lives under inferencex-e2e/. Those local checks produced the following results.

Recorded check Result What it tells you
Install Succeeded in 10 seconds; 41 dependencies, 48 MB on disk The recorded dependency step completed
Build Succeeded in 4 seconds The recorded build step completed
Tests Exit 1 after 232 seconds; 2,767 passed, 47 failed, 153 collection/setup errors, total 2,967 The test command stayed red despite many passing cases
pip-audit 0 known vulnerabilities No known advisories reported by that audit
Repository scan 29 CI workflow files; no Dockerfile and no tests directory detected A layout scan, not a verdict on CI health

The failed log tail repeatedly names launch snapshot cases under utils/srt-slurm/tests/test_launch_snapshots.py, ending with ModuleNotFoundError: No module named 'marshmallow'. Those nested test paths also explain why the scanner's "no tests directory" label must not become a claim that the project has no tests. The visible tail gives one concrete import failure; it supplies no basis for assigning all 47 failures or 153 errors to that module.

The outer test step took 232 seconds, while pytest's own summary reports 227.31 seconds. The recorded image is ghcr.io/astral-sh/uv:python3.12-bookworm; actual runtime and operating-system versions require raw toolchain logs. A quick successful build is useful to a contributor, but the red suite means this checkout needs more investigation before it can serve as your release gate. It supplies no GPU performance result for the charts.

Where people get stuck: orchestration and missing evidence

Issue #870 shows a benchmark completing while the requested power CSV collection was missing. The custom workflow could not be dispatched from the initial branch, and a fallback used existing infrastructure without the monitoring. Later work integrated monitoring and explicitly selected branches containing it. If you want watts alongside tokens, inspect the artifact list before spending time interpreting the run. Monitoring thread.

Symptom in the records Reported response Outcome to carry forward
B200 reproduction falls short and historical logs have expired Maintainer asks for Slurm reproduction, then Kubernetes environment parity Inactivity closure; no successful reproduction in the thread
Serving benchmark finishes without requested GPU telemetry Add monitoring to the benchmark infrastructure and use the matching branch Later B200 analysis reports benchmark and power results
Same MI355X recipe behaves differently between nodes Investigate host bursts, CPU idle behavior and governor Final report says latency limits plus performance governor removed host slowdown on treated nodes
AgentX cold preparation appears idle Pin corpus and seed for cache reuse; raise both configuration timeouts when needed Documented operating guidance, not a measured fix from our sandbox

The final row comes from the AgentX guide. Its preparation phase reconstructs conversations before the timed profile begins. Killing a quiet-looking preparation step can waste work that would have been reusable. Pin the dataset and random seed, retain the generated artifacts, and distinguish preparation, warmup and profiling when reading a run's elapsed time.

Host CPU settings belong in your reproduction checklist

In issue #4339, investigators reported different AgentX performance between MI355X nodes running the same GLM-5.2 ATOM recipe. An intermediate comment said a power cycle fixed an IPC probe. A later benchmark still ran slowly. Investigator seungrokj attributes the host-burst slowdown to BIOS 1.7a idle/P-state behavior. The final update reports applying CPU idle latency limits together with the performance governor, with the host-burst slowdown gone on the treated nodes. Two nodes were excluded from that treatment. Host investigation.

That sequence matters more than a broad claim that rebooting helps. For a reproduction, record the serving image and recipe, accelerator topology, host BIOS, CPU governor and idle policy. Add the router policy and load-generator placement. Check client CPU saturation and connection-pool waiting as the official guide recommends. A server benchmark can become a client benchmark if the load generator runs out of capacity. Client diagnostics.

For multi-replica AgentX, preserve conversation affinity. The docs explain that scattering turns across replicas destroys prefix reuse. Prefix-aware routing or sticky conversation headers therefore belong in the saved configuration. Comparing two recipes while silently changing routing can answer a different question from the one you meant to ask. Routing requirements.

A curve can contain several configurations

An AgentX curve can combine different optimizations at different points, including speculative decoding, disaggregation and cache offload. The telemetry tutorial explicitly describes this construction. Choose a point, then inspect its precision, image, parallelism, workload and provenance. Reading the entire curve as one configuration would give an operator a misleading deployment target. How curves are built.

Power needs equally careful units. The API examples define average watts per GPU separately from whole-deployment GPU energy per output token. Neither quantity is facility energy or a provisioned-power estimate. Shared prefill and decode GPUs must not be double-counted. Leave missing power fields missing, and compare costs at an explicit interactivity target with declared prices. A cheap saturated configuration may be the wrong choice for the response speed your users need. Power and cost definitions.

Save each observation's measurement date as well as the retrieval date. The API docs say that even an exact snapshot can contain older observations. They also say deprecated models remain queryable. Record workload selectors and model status alongside the exported result. This is how you keep a fresh download from becoming a claim about a fresh measurement.

Security starts at the server and tokenizer boundary

The standalone path requires a reachable model endpoint and resolvable tokenizer; NVIDIA documents bearer authentication when the server requires it and access credentials for gated tokenizers. The official quickstart also includes a tokenizer remote-code trust flag. Review that choice against the tokenizer source you intend to use, and keep benchmark credentials scoped to the endpoint. Public recipes do not take over those operating decisions. Execution prerequisites, official quickstart.

The workload uses synthesized prompt text while preserving recorded session structure. Its purpose is to exercise serving behavior without replaying private user content. This also means you should not read the generated prompt prose as a coding-quality test. The SGLang optimization page describes a cache-staging correctness failure that went from 2 correct needles out of 128 to 128 out of 128 after a fix. Throughput alone would have missed the wrong answers. Workload handling, correctness report.

InferenceX vs AIPerf and metrumbench-llm

Use InferenceX for published recipes, AIPerf for direct AgentX replay, and metrumbench-llm for prompt-library or ramp-up experiments. These choices overlap with pieces of InferenceX, so select the experiment before selecting the repository. NVIDIA documents running the AgentX scenario in AIPerf against an OpenAI-compatible server. Metrum recommends metrumbench-llm for prompt libraries and ramp-up analysis. Metrum's comparison describes its own fixed-length InferenceX integration. AIPerf scenario, Metrum comparison.

Your question Better starting point Why
How do published hardware and engine configurations compare? InferenceX observations and recipes Compare workload-specific serving results with provenance
How does my endpoint handle dependent coding-agent sessions? AIPerf / AgentX Replay shared prefixes, timing and subagent relationships
How does my prompt library behave during ramp-up or behind replicas? metrumbench-llm within Metrum AI Bench Metrum recommends it for these traffic and client-metric needs
What is baseline chip/kernel behavior without session reuse? Fixed-sequence serving test Independent requests remove the agentic prefix-reuse dimension

The last distinction comes from SemiAnalysis's workload explanation. Metrum's restrictions apply to its integration, not to every InferenceX workload. AgentX itself documents multi-replica routing. None of these sources supplies a same-hardware contest between all the tools, so there is no supported speed winner to put in this table.

What the other reviews get wrong

Silicon Data's comparison says the public cannot run InferenceX on its own systems, whereas the official About page provides an own-chip AgentX quickstart. NVIDIA also documents execution against a reachable OpenAI-compatible endpoint. Public execution is documented; owning the infrastructure for a full benchmark remains your responsibility. Silicon Data comparison, execution guide.

Its fixed-length workload description is also incomplete for AgentX. The later official material describes long-context multi-turn replay, and issue #2287 records the maintainer's explanation for retiring the 1K/1K and 1K/8K single-turn scenarios. The v2 announcement's future-tense Ironwood roadmap has aged too: the supplied dashboard lists TPUv7 Ironwood, and SemiAnalysis's September post announces daily TPU benchmarking. Scenario discussion, dashboard, announcement records.

Two official descriptions need care as well. Public provenance has an expiry boundary: the About page specifies 90-day GitHub artifact retention. NVIDIA warns that changed trajectory windows can still receive submission_valid: true. Reducing the eligible corpus can also change the workload without invalidating the stamp. Save the actual settings and artifacts. A validity stamp alone cannot tell you two workloads are equivalent. Retention policy, scenario checks.

Questions people ask

InferenceX is SemiAnalysis's platform for continuously comparing fixed-sequence and agentic inference across hardware and serving stacks. Official description.

What is InferenceX?

SemiAnalysis maintains InferenceX, formerly InferenceMAX, as an open-source inference performance research platform. It includes benchmark tooling and public results; AgentX is its long-context, multi-turn scenario. About InferenceX.

Where is the official InferenceX GitHub repository?

The official InferenceX GitHub repository is SemiAnalysisAI/InferenceX. Its README distinguishes official results from forks, and the former InferenceMAX repository points to this address. The dashboard code lives separately in SemiAnalysisAI/InferenceX-app. Repository move.

Does InferenceX offer long-output comparisons such as 1K input and 64K output tokens?

Issue #2287 contains that exact request, but the supplied discussion gives no confirmation that a 1K/64K scenario was accepted. The maintainer instead explains the transition to AgentX and the limits of available GPU resources. Long-output request.

Why were the 1K/1K and 1K/8K single-turn scenarios deprecated?

A maintainer in issue #2287 says submitters agreed to move to long-context multi-turn workloads incorporating cache hits and agentic coding. That explains the scenario change; it supplies no evidence for a commenter's allegation that the choice favored a hardware vendor. Maintainer response.

What does the Agentic Traces benchmark actually measure?

AgentX measures serving under dependent sessions with growing context, prefix reuse, pauses and subagent fan-out. The prompt text is synthesized while the session structure comes from recordings. It measures the serving workload rather than whether a model completes the original coding task. AgentX operating model.

Can an in-house inference engine participate, or are submissions limited to vLLM, SGLang and TensorRT-LLM?

An OpenAI-compatible endpoint can be exercised through the documented AgentX client. Acceptance into hosted official coverage is a separate question: in issue #2287 the maintainer cites engineering and cluster time as reasons for limiting engines. The thread promises no automatic inclusion for an arbitrary in-house engine. Execution prerequisites, coverage discussion.

Why are AgentX outputs short if the traces represent coding tasks?

A turn represents one model request within an agentic loop, rather than the whole coding task, according to the maintainer in issue #2287. Compare the accumulated conversation and cache use as well as individual output lengths. Output-length explanation.

How can I reproduce an InferenceX result after its GitHub Actions logs expire?

Issue #1691 shows that an expired-log request can end without restored evidence or a successful reproduction. Preserve artifacts while available and use the committed recipe as a starting point. That thread's maintainer recommends Slurm first, then checking Kubernetes environment parity. Reproduction discussion.

How many frontends and load-generator clients were used for a published multi-node result?

OrZipori asked both questions for the B200 result in issue #1691, and the supplied thread ends without answers. Do not infer either count from the GPU total. Obtain the run-specific deployment evidence before claiming a matched reproduction. Unanswered deployment questions.

Why place prefill and decode workers on separate nodes instead of mixing them?

The B200 user asks about that placement in issue #1691, but the supplied replies provide no rationale. The thread therefore supports no general claim that separate nodes are always better for KV transfer. Treat placement as part of the recipe to investigate. Placement question.

When will InferenceX add accuracy evaluations beyond GSM8K?

Benjamin Goel asks for broader accuracy evaluation in the supplied SemiAnalysis LinkedIn discussion. That record contains no reply giving a schedule. Keep a throughput result separate from a claim about model accuracy. Accuracy question.

How does average GPU power change with interactivity and concurrency?

Issue #828 reports collecting power CSVs across tensor parallelism, concurrency and input/output shapes, then expanding an initially TP8-focused analysis. It supports examining those dimensions together, rather than applying one power number to every operating point. Use the matching artifacts for a particular configuration. Power experiment.

What we have not tested yet

Our evidence covers the November 4 source sandbox and a November 13 hands-on attempt blocked before deployment. We have not reproduced a GPU sweep, a published cost curve, power readings or an AgentX session against a live model server. No successful live task or screenshot was recorded in that attempt.

The marshmallow import error needs complete logs and a rerun before a package-level remedy can be prescribed. Our pip-audit result covers known dependency advisories, not tokenizer code, endpoint access or cluster permissions. The B200 report remains unresolved, and the CPU investigation's final treatment excludes two nodes. Those community reports are useful reproduction evidence, not hardware performance we measured ourselves.

All four card scores are unassessed. Source checks do not assess supported accelerator setup, selected issue threads do not measure support capacity, and reading instructions cannot establish their success across the hardware matrix. For your first useful result, choose one dashboard point, archive its evidence, and run that exact workload on the deployment you would actually operate.

Alternatives

ProjectWhat it isPick it when
AIPerf gh↗A benchmark client with a documented AgentX scenario for OpenAI-compatible servers.Choose the direct client when you already have a server and want to replay AgentX without the full repository orchestration.
metrumbench-llm within Metrum AI BenchMetrum documents prompt-library traffic, ramp-up analysis and client-side streaming metrics.Choose this path when prompt diversity or multi-replica endpoint testing is the experiment you need.

Sources

  1. Official InferenceX repository
  2. AgentX methodology and reproducibility
  3. NVIDIA AgentX execution guide
  4. Metrum benchmarking comparison
  5. B200 reproduction thread
  6. Host CPU investigation
  7. Power and API definitions
  8. AgentX curve construction

More ai tools reviews