AIPerf measures the client view across many inference workloads
AIPerf sends traffic to an already running model endpoint and records what the client observes. For streamed text that includes time to first token, inter-token latency, output throughput, request latency, and sequence lengths. Supported endpoint families extend beyond chat to completions, embeddings, rankings, audio, images, and OpenAI Responses. Results can appear in a live terminal dashboard, simpler progress display, or headless run with CSV and JSON artifacts.
The useful part is workload control. A fixed concurrency test answers a different question from constant or probabilistic arrivals. AIPerf supports request rate with a concurrency ceiling, warmup phases, gradual ramps, fixed schedules, cancellations, multi-URL routing, and repeated runs with confidence analysis. Trace replay can preserve production timing, while synthetic distributions let an operator vary input length, output length, and conversation turns without sharing real prompts.
The 10-service design is built for experiments larger than a smoke test
The README describes a multiprocess architecture with 10 services communicating through ZeroMQ. Plugins cover endpoints, datasets, transports, and metrics. That separation allows custom request formats and measurement logic, but it also explains the dependency footprint and the amount of code. A one-line curl loop is easier to inspect; AIPerf is for cases where pacing, data generation, recording, and analysis must remain consistent across many runs.
Configuration can live in YAML rather than a long shell command. Sampling distributions include fixed, normal, log-normal, multimodal, and empirical shapes. Public datasets and exact custom payloads can drive the workload. Teams can replay SageMaker captures or supported trace formats, preserve a random seed, and use local tokenizers when downloading from Hugging Face is undesirable. These controls make comparisons more defensible, provided every server sees the same requests.
What happened when we ran it
Our run at commit d8d49e8 used a fresh unprivileged Python 3.12 Bookworm container with 3 CPUs, 8 GB of RAM, and no secrets. Installation succeeded in 59 seconds, adding 207 packages and consuming 1,038 MB on disk. The build then passed in 8 seconds. pip-audit reported 21 known vulnerabilities in the resulting environment.
The test command failed with exit 1 after 15 seconds. Pytest completed 177 tests with 0 assertion failures and 1 warning, but it raised 8 collection or setup errors out of 185. The log names several mock-server test modules plus component, integration, and unit directories, all ending with ModuleNotFoundError: No module named 'aiperf_mock_server'. We cannot call that a passing suite just because the tests that started were green.
The checkout contained 2,361 files, about 599,768 source lines, and 41.4 MB before installation. Our scan found 10 CI workflow files, a Dockerfile, and a tests directory. Those are good engineering signals, while the measured import gap remains a concrete contributor problem on this commit and environment. The log does not establish whether packaging, test setup, or another configuration caused it, so we will not guess.
A benchmark is only fair when the workload and tokenizer match
AIPerf needs a model name, endpoint type, URL, and tokenizer for its basic profile. Protected services add credentials. Optional MLflow, OpenTelemetry, and Weights & Biases integrations add packages plus remote configuration. The quick start also launches Ollama in Docker and pulls a model, which is setup for the target server rather than for AIPerf itself. Our 59-second install did not include model weights or a real inference run.
Token accounting can quietly ruin a comparison. Use the tokenizer that matches the served model, keep prompt data fixed, record whether streaming is enabled, and separate warmup from the measured phase. The README warns that requested output length cannot be guaranteed unless the inference server accepts controls such as ignore_eos or min_tokens. If two servers stop generation differently, their throughput and latency figures are not directly comparable.
Client placement matters too. DNS, TCP, TLS, queues, routers, and network distance become part of observed latency. AIPerf can record HTTP trace metrics and scrape Prometheus-compatible server metrics, but the report still needs the topology beside the numbers. Run the load generator away from the server only when that path represents users, and monitor whether the 207-package client itself reaches CPU, port, or memory limits.
Known limits are specific enough to plan around
The README says very high concurrency, typically above 15,000, may exhaust ports on some systems. Invalid configuration can also leave startup hanging, requiring termination and a review of settings. Dashboard text selection may be unreliable, with a key command offered to copy logs. These are unusually candid constraints and should become preflight checks in an automated benchmark job.
AIPerf also supports prefill concurrency for long contexts, goodput against service-level objectives, parameter sweeps, adaptive search, and timeslice analysis. These features can answer where a configuration stops meeting a latency target, not merely which run produced the largest throughput number. They also increase the number of choices. Start with one controlled mode and add sweeps only after its artifacts and metrics agree with direct server observations.
August activity is fast enough to require version pinning
GitHub recorded a push on 2026-08-26 and 91 open issues and pull requests combined. Same-day work covered Kubernetes execution, configuration handling, structured streaming errors, dataset validation, metrics, and accuracy. Release v0.12.0 was published on 2026-08-06 and dropped Python 3.10 support, making Python 3.11 the minimum. Linux arm64 users may also need a C compiler for crick.
That pace is healthy for new inference workloads, but benchmark methodology should not change unnoticed between reports. Pin the AIPerf version, configuration, dataset fingerprint, tokenizer, server build, hardware, and topology with every result. AIPerf provides enough controls to make repeatable experiments. It cannot supply the discipline or decide whether a 2% movement is software, traffic, or measurement noise.