Short answer: use the charts to choose a recipe to reproduce
Our InferenceX sandbox built in 4 seconds, then its test command failed with 47 failures and 153 collection/setup errors. Use the project if you run inference infrastructure and can investigate the exact configuration behind a promising result. Skip it if you need a ready-made hardware buying recommendation: changing the workload, routing or GPU denominator changes the comparison you are making.
InferenceX is SemiAnalysis's open-source inference benchmarking platform, formerly called InferenceMAX. Its useful output is a comparison tied to a model, serving engine and workload. Start with one point that meets your response-speed target, save the configuration and artifacts, and reproduce that point on your own deployment. The public records show why reproduction matters: a published recipe can leave important deployment questions unanswered. Platform description, reproduction report.
The benchmark, dashboard and client have separate jobs
The InferenceX repository runs benchmarks, the dashboard displays results, and API skills query existing observations. The AgentX client replays traffic against a model server. These components answer different questions and have different setup requirements. Reading public results starts with the dashboard; replaying a workload starts with a reachable serving endpoint; operating your own dashboard adds its database. Choose the component that matches the work you intend to do.
InferenceX compares fixed-sequence serving and AgentX, a long-context workload that replays dependent coding-agent sessions. The distinction changes what a result means. Independent requests expose baseline chip and kernel performance. AgentX brings shared prefixes, growing conversations, pauses and subagents into the measurement, so routing and cache behavior affect the outcome. Pick the scenario that resembles the traffic you intend to serve. Workload explanation.
| Component | Job | What you provide |
|---|---|---|
| SemiAnalysisAI/InferenceX | Benchmark recipes and execution tooling | Serving infrastructure for the selected recipe |
| SemiAnalysisAI/InferenceX-app | Dashboard and database-backed visualization | PostgreSQL when hosting your own copy |
| AgentX harness / AIPerf scenario | Replay traffic against a model server | Reachable OpenAI-compatible endpoint and tokenizer |
| InferenceX API skills | Query existing observations | Public HTTPS access; queries do not launch benchmarks |
The official benchmark README links a standalone AgentX route that avoids CI and Slurm. The dashboard README and API examples describe the other two paths. Start with the public dashboard if your question is about existing results. Host the application only when you need to operate that application yourself.
GitHub's public API returned 2,050 stars, 171 open issues and PRs, and a last push on 17 November 2026. The sampled tracker includes both detailed investigations and an unresolved reproduction request closed for inactivity. That combination supports calling the repository active; it does not establish a support commitment or widespread successful reproduction. The latest-release endpoint returned h3-v2-release-v1-references, published on 7 November. Read that reference tag alongside current commits rather than treating a release tag as a complete account of project health. Public repository metadata.
What people use InferenceX for
The issue records show engineers using InferenceX for reproduction attempts and instrumented serving experiments, with outcomes that need reading beyond the issue's closed label.
- GitHub user OrZipori attempted a DeepSeek Dynamo TRT-LLM plus MTP reproduction on four B200 SXM nodes using Kubernetes. OrZipori writes, "Current best result is ~8.3K per decode gpu." The referenced published result was 10,012 tokens/s/GPU. Maintainers suggested trying Slurm first; the thread closed for inactivity without a successful reproduction. OrZipori reports throughput per decode GPU, while the published result is labeled per GPU; the thread also contrasts Kubernetes with Slurm. Those differences prevent treating the gap as a matched regression. Issue #1691.
- functionstackx asked the repository's Claude automation to sweep gpt-oss-120b FP4 on H100 with power collection. The automation reported 22 power CSVs plus benchmark results. When its first analysis focused on TP8, functionstackx requested coverage of all tensor-parallel configurations. That correction is a useful reminder to inspect what an automated analysis actually included. Issue #828.
- functionstackx also requested DeepSeek SGLang sweeps with GPU monitoring. An early fallback run omitted the requested telemetry; later branch-specific B200 work reported five successful configurations and analysis of benchmark and GPU power results. A completed serving run and a completed instrumentation task are separate outcomes. Issue #870.
Outside the tracker, Together researcher simran_s_arora claims its compiler beat the best InferenceX engine across the settings it tested. BackboardIO also posts comparisons against selected InferenceX baselines. These are vendor-authored comparisons with selected workloads, rather than our measurements or independent confirmation. They show how the benchmark is used as a reference point, not how many production deployments depend on it. Collected X posts.
What happened when we ran it
Our November 13 hands-on attempt stopped at a safety check because the repository had none of the Compose filenames required by that runner. The record calls it an "unsafe repo," but the reason shown is the runner's Compose requirement. That label is no finding about malicious code. Teardown reported success. A generic run summary says it deployed with Docker Compose and loopback ports; the failed step contradicts that summary, so we count this as a blocked attempt.
Our separate source sandbox ran commit 3b4b94f on November 4, 2026, with 3 CPUs and 8 GB of RAM. The Python project lives under inferencex-e2e/. Those local checks produced the following results.
| Recorded check | Result | What it tells you |
|---|---|---|
| Install | Succeeded in 10 seconds; 41 dependencies, 48 MB on disk | The recorded dependency step completed |
| Build | Succeeded in 4 seconds | The recorded build step completed |
| Tests | Exit 1 after 232 seconds; 2,767 passed, 47 failed, 153 collection/setup errors, total 2,967 | The test command stayed red despite many passing cases |
| pip-audit | 0 known vulnerabilities | No known advisories reported by that audit |
| Repository scan | 29 CI workflow files; no Dockerfile and no tests directory detected | A layout scan, not a verdict on CI health |
The failed log tail repeatedly names launch snapshot cases under utils/srt-slurm/tests/test_launch_snapshots.py, ending with ModuleNotFoundError: No module named 'marshmallow'. Those nested test paths also explain why the scanner's "no tests directory" label must not become a claim that the project has no tests. The visible tail gives one concrete import failure; it supplies no basis for assigning all 47 failures or 153 errors to that module.
The outer test step took 232 seconds, while pytest's own summary reports 227.31 seconds. The recorded image is ghcr.io/astral-sh/uv:python3.12-bookworm; actual runtime and operating-system versions require raw toolchain logs. A quick successful build is useful to a contributor, but the red suite means this checkout needs more investigation before it can serve as your release gate. It supplies no GPU performance result for the charts.
Where people get stuck: orchestration and missing evidence
Issue #870 shows a benchmark completing while the requested power CSV collection was missing. The custom workflow could not be dispatched from the initial branch, and a fallback used existing infrastructure without the monitoring. Later work integrated monitoring and explicitly selected branches containing it. If you want watts alongside tokens, inspect the artifact list before spending time interpreting the run. Monitoring thread.
| Symptom in the records | Reported response | Outcome to carry forward |
|---|---|---|
| B200 reproduction falls short and historical logs have expired | Maintainer asks for Slurm reproduction, then Kubernetes environment parity | Inactivity closure; no successful reproduction in the thread |
| Serving benchmark finishes without requested GPU telemetry | Add monitoring to the benchmark infrastructure and use the matching branch | Later B200 analysis reports benchmark and power results |
| Same MI355X recipe behaves differently between nodes | Investigate host bursts, CPU idle behavior and governor | Final report says latency limits plus performance governor removed host slowdown on treated nodes |
| AgentX cold preparation appears idle | Pin corpus and seed for cache reuse; raise both configuration timeouts when needed | Documented operating guidance, not a measured fix from our sandbox |
The final row comes from the AgentX guide. Its preparation phase reconstructs conversations before the timed profile begins. Killing a quiet-looking preparation step can waste work that would have been reusable. Pin the dataset and random seed, retain the generated artifacts, and distinguish preparation, warmup and profiling when reading a run's elapsed time.
Host CPU settings belong in your reproduction checklist
In issue #4339, investigators reported different AgentX performance between MI355X nodes running the same GLM-5.2 ATOM recipe. An intermediate comment said a power cycle fixed an IPC probe. A later benchmark still ran slowly. Investigator seungrokj attributes the host-burst slowdown to BIOS 1.7a idle/P-state behavior. The final update reports applying CPU idle latency limits together with the performance governor, with the host-burst slowdown gone on the treated nodes. Two nodes were excluded from that treatment. Host investigation.
That sequence matters more than a broad claim that rebooting helps. For a reproduction, record the serving image and recipe, accelerator topology, host BIOS, CPU governor and idle policy. Add the router policy and load-generator placement. Check client CPU saturation and connection-pool waiting as the official guide recommends. A server benchmark can become a client benchmark if the load generator runs out of capacity. Client diagnostics.
For multi-replica AgentX, preserve conversation affinity. The docs explain that scattering turns across replicas destroys prefix reuse. Prefix-aware routing or sticky conversation headers therefore belong in the saved configuration. Comparing two recipes while silently changing routing can answer a different question from the one you meant to ask. Routing requirements.
A curve can contain several configurations
An AgentX curve can combine different optimizations at different points, including speculative decoding, disaggregation and cache offload. The telemetry tutorial explicitly describes this construction. Choose a point, then inspect its precision, image, parallelism, workload and provenance. Reading the entire curve as one configuration would give an operator a misleading deployment target. How curves are built.
Power needs equally careful units. The API examples define average watts per GPU separately from whole-deployment GPU energy per output token. Neither quantity is facility energy or a provisioned-power estimate. Shared prefill and decode GPUs must not be double-counted. Leave missing power fields missing, and compare costs at an explicit interactivity target with declared prices. A cheap saturated configuration may be the wrong choice for the response speed your users need. Power and cost definitions.
Save each observation's measurement date as well as the retrieval date. The API docs say that even an exact snapshot can contain older observations. They also say deprecated models remain queryable. Record workload selectors and model status alongside the exported result. This is how you keep a fresh download from becoming a claim about a fresh measurement.
Security starts at the server and tokenizer boundary
The standalone path requires a reachable model endpoint and resolvable tokenizer; NVIDIA documents bearer authentication when the server requires it and access credentials for gated tokenizers. The official quickstart also includes a tokenizer remote-code trust flag. Review that choice against the tokenizer source you intend to use, and keep benchmark credentials scoped to the endpoint. Public recipes do not take over those operating decisions. Execution prerequisites, official quickstart.
The workload uses synthesized prompt text while preserving recorded session structure. Its purpose is to exercise serving behavior without replaying private user content. This also means you should not read the generated prompt prose as a coding-quality test. The SGLang optimization page describes a cache-staging correctness failure that went from 2 correct needles out of 128 to 128 out of 128 after a fix. Throughput alone would have missed the wrong answers. Workload handling, correctness report.
InferenceX vs AIPerf and metrumbench-llm
Use InferenceX for published recipes, AIPerf for direct AgentX replay, and metrumbench-llm for prompt-library or ramp-up experiments. These choices overlap with pieces of InferenceX, so select the experiment before selecting the repository. NVIDIA documents running the AgentX scenario in AIPerf against an OpenAI-compatible server. Metrum recommends metrumbench-llm for prompt libraries and ramp-up analysis. Metrum's comparison describes its own fixed-length InferenceX integration. AIPerf scenario, Metrum comparison.
| Your question | Better starting point | Why |
|---|---|---|
| How do published hardware and engine configurations compare? | InferenceX observations and recipes | Compare workload-specific serving results with provenance |
| How does my endpoint handle dependent coding-agent sessions? | AIPerf / AgentX | Replay shared prefixes, timing and subagent relationships |
| How does my prompt library behave during ramp-up or behind replicas? | metrumbench-llm within Metrum AI Bench | Metrum recommends it for these traffic and client-metric needs |
| What is baseline chip/kernel behavior without session reuse? | Fixed-sequence serving test | Independent requests remove the agentic prefix-reuse dimension |
The last distinction comes from SemiAnalysis's workload explanation. Metrum's restrictions apply to its integration, not to every InferenceX workload. AgentX itself documents multi-replica routing. None of these sources supplies a same-hardware contest between all the tools, so there is no supported speed winner to put in this table.
What the other reviews get wrong
Silicon Data's comparison says the public cannot run InferenceX on its own systems, whereas the official About page provides an own-chip AgentX quickstart. NVIDIA also documents execution against a reachable OpenAI-compatible endpoint. Public execution is documented; owning the infrastructure for a full benchmark remains your responsibility. Silicon Data comparison, execution guide.
Its fixed-length workload description is also incomplete for AgentX. The later official material describes long-context multi-turn replay, and issue #2287 records the maintainer's explanation for retiring the 1K/1K and 1K/8K single-turn scenarios. The v2 announcement's future-tense Ironwood roadmap has aged too: the supplied dashboard lists TPUv7 Ironwood, and SemiAnalysis's September post announces daily TPU benchmarking. Scenario discussion, dashboard, announcement records.
Two official descriptions need care as well. Public provenance has an expiry boundary: the About page specifies 90-day GitHub artifact retention. NVIDIA warns that changed trajectory windows can still receive submission_valid: true. Reducing the eligible corpus can also change the workload without invalidating the stamp. Save the actual settings and artifacts. A validity stamp alone cannot tell you two workloads are equivalent. Retention policy, scenario checks.
Questions people ask
InferenceX is SemiAnalysis's platform for continuously comparing fixed-sequence and agentic inference across hardware and serving stacks. Official description.
What is InferenceX?
SemiAnalysis maintains InferenceX, formerly InferenceMAX, as an open-source inference performance research platform. It includes benchmark tooling and public results; AgentX is its long-context, multi-turn scenario. About InferenceX.
Where is the official InferenceX GitHub repository?
The official InferenceX GitHub repository is SemiAnalysisAI/InferenceX. Its README distinguishes official results from forks, and the former InferenceMAX repository points to this address. The dashboard code lives separately in SemiAnalysisAI/InferenceX-app. Repository move.
Does InferenceX offer long-output comparisons such as 1K input and 64K output tokens?
Issue #2287 contains that exact request, but the supplied discussion gives no confirmation that a 1K/64K scenario was accepted. The maintainer instead explains the transition to AgentX and the limits of available GPU resources. Long-output request.
Why were the 1K/1K and 1K/8K single-turn scenarios deprecated?
A maintainer in issue #2287 says submitters agreed to move to long-context multi-turn workloads incorporating cache hits and agentic coding. That explains the scenario change; it supplies no evidence for a commenter's allegation that the choice favored a hardware vendor. Maintainer response.
What does the Agentic Traces benchmark actually measure?
AgentX measures serving under dependent sessions with growing context, prefix reuse, pauses and subagent fan-out. The prompt text is synthesized while the session structure comes from recordings. It measures the serving workload rather than whether a model completes the original coding task. AgentX operating model.
Can an in-house inference engine participate, or are submissions limited to vLLM, SGLang and TensorRT-LLM?
An OpenAI-compatible endpoint can be exercised through the documented AgentX client. Acceptance into hosted official coverage is a separate question: in issue #2287 the maintainer cites engineering and cluster time as reasons for limiting engines. The thread promises no automatic inclusion for an arbitrary in-house engine. Execution prerequisites, coverage discussion.
Why are AgentX outputs short if the traces represent coding tasks?
A turn represents one model request within an agentic loop, rather than the whole coding task, according to the maintainer in issue #2287. Compare the accumulated conversation and cache use as well as individual output lengths. Output-length explanation.
How can I reproduce an InferenceX result after its GitHub Actions logs expire?
Issue #1691 shows that an expired-log request can end without restored evidence or a successful reproduction. Preserve artifacts while available and use the committed recipe as a starting point. That thread's maintainer recommends Slurm first, then checking Kubernetes environment parity. Reproduction discussion.
How many frontends and load-generator clients were used for a published multi-node result?
OrZipori asked both questions for the B200 result in issue #1691, and the supplied thread ends without answers. Do not infer either count from the GPU total. Obtain the run-specific deployment evidence before claiming a matched reproduction. Unanswered deployment questions.
Why place prefill and decode workers on separate nodes instead of mixing them?
The B200 user asks about that placement in issue #1691, but the supplied replies provide no rationale. The thread therefore supports no general claim that separate nodes are always better for KV transfer. Treat placement as part of the recipe to investigate. Placement question.
When will InferenceX add accuracy evaluations beyond GSM8K?
Benjamin Goel asks for broader accuracy evaluation in the supplied SemiAnalysis LinkedIn discussion. That record contains no reply giving a schedule. Keep a throughput result separate from a claim about model accuracy. Accuracy question.
How does average GPU power change with interactivity and concurrency?
Issue #828 reports collecting power CSVs across tensor parallelism, concurrency and input/output shapes, then expanding an initially TP8-focused analysis. It supports examining those dimensions together, rather than applying one power number to every operating point. Use the matching artifacts for a particular configuration. Power experiment.
What we have not tested yet
Our evidence covers the November 4 source sandbox and a November 13 hands-on attempt blocked before deployment. We have not reproduced a GPU sweep, a published cost curve, power readings or an AgentX session against a live model server. No successful live task or screenshot was recorded in that attempt.
The marshmallow import error needs complete logs and a rerun before a package-level remedy can be prescribed. Our pip-audit result covers known dependency advisories, not tokenizer code, endpoint access or cluster permissions. The B200 report remains unresolved, and the CPU investigation's final treatment excludes two nodes. Those community reports are useful reproduction evidence, not hardware performance we measured ourselves.
All four card scores are unassessed. Source checks do not assess supported accelerator setup, selected issue threads do not measure support capacity, and reading instructions cannot establish their success across the hardware matrix. For your first useful result, choose one dashboard point, archive its evidence, and run that exact workload on the deployment you would actually operate.


