NVIDIA releases AIPerf: a multiprocess, scalable LLM benchmarking client

NVIDIA has released AIPerf, a ground-up successor to GenAI-Perf designed to benchmark large language model (LLM) endpoints at production-like scale. The tool aims to avoid common client-side bottlenecks, provide actionable, reproducible metrics, and be quick to configure.

What AIPerf changes

• Architectural break: AIPerf is a full rewrite rather than an evolution on Perf Analyzer. That clean break enables different scaling choices and behaviors; a migration guide documents the key deltas for existing workflows.

• Multiprocess client design: unlike many benchmarkers that run in a single process and become GIL-bound under real concurrency, AIPerf uses multiple worker processes to generate load, separate record-processor services to handle results, and ZMQ for coordination. This reduces the likelihood that the client itself limits measured server performance.

• Workload breadth: AIPerf supports more than 15 endpoint types (chat, responses, NIM rankings, image generation, and more) and accepts public datasets and trace-replay formats such as ShareGPT, Mooncake, Baseten, and WEKA (AgentX). You can run quick synthetic smoke tests or replay captured production traffic with the same tool.

• Controllable load shapes: AIPerf supports constant, Poisson, and gamma arrival patterns with tunable burstiness, gradual ramping for concurrency and request rate, and synthetic distributions (including vLLM/SGLang range-ratio). You control the shape of incoming traffic, not just its volume.

Hands-on: synthetic ISL/OSL measurement on vLLM

The walkthrough uses Qwen/Qwen3-0.6B served via vLLM. The model is small enough to run on a single GPU and fast enough for iterative testing; the purpose is to establish the measurement loop so you can substitute a different model or endpoint with a single flag change.

Start the server

Pull and run vLLM with the reasoning parser enabled:

docker pull vllm/vllm-openai:latest
docker run --gpus all -p 8000:8000 -e HF_TOKEN vllm/vllm-openai:latest \
  --model Qwen/Qwen3-0.6B \
  --reasoning-parser qwen3 \
  --host 0.0.0.0 --port 8000

Install AIPerf

Central install via uv:

uv tool install aiperf

Or in a virtual environment:

uv venv venv
source venv/bin/activate
uv pip install aiperf

Platform note: on aarch64, the crick dependency ships source-only and requires a C toolchain (e.g., build-essential on Debian/Ubuntu). If installation stalls on that package, this is likely the cause.

Run the benchmark

A simple static profile example:

aiperf profile \
  --model Qwen/Qwen3-0.6B \
  --endpoint-type chat \
  --streaming \
  --url localhost:8000 \
  --synthetic-input-tokens-mean 128 \
  --synthetic-input-tokens-stddev 0 \
  --output-tokens-mean 128 \
  --output-tokens-stddev 0 \
  --extra-inputs min_tokens:128 \
  --extra-inputs ignore_eos:true

Key flags and their effects:

• --synthetic-input-tokens-stddev 0 and --output-tokens-stddev 0 pin input and output lengths to exactly 128 tokens, producing a static, reproducible baseline.
• --extra-inputs min_tokens:128 and ignore_eos:true force the model to emit 128 tokens rather than stopping early, ensuring output token counts are reproducible.
• --streaming is mandatory for measuring TTFT and ITL; without streaming the server sends the whole response at once and there are no first- or decode-token events to capture.

What you’ll see

AIPerf’s live dashboard displays run progress, metric distributions, and an event log. At run completion AIPerf prints a metrics table to the console and writes full results to CSV and JSON.

Core metrics AIPerf reports

• TTFT (Time to First Token): time from request dispatch to first token received — the primary latency signal for interactive use cases.
• ITL (Inter-Token Latency): time between successive tokens during generation — high ITL indicates decode-phase issues even if TTFT looks fine.
• Request Latency: end-to-end time for the full response, combining prefill and decode cost.
• Output Token Throughput: tokens generated per second across all concurrent requests — the primary throughput signal for capacity planning.

Each metric is reported with percentile breakdowns (p25, p50, p75, p90, p95, p99) as well as min, max, mean and standard deviation. Those percentiles are important because a server with a healthy mean but a poor p99 can still fail in production.

GPU telemetry

If DCGM or pynvml is available, AIPerf collects GPU power draw, utilization and memory consumption alongside the metrics, making it easier to correlate latency spikes with resource pressure.

Configuring dynamic traffic patterns

To model realistic, non-static traffic, you can use Poisson arrival patterns and variable prompt/output lengths. Example:

aiperf profile \
  --model Qwen/Qwen3-0.6B \
  --endpoint-type chat \
  --streaming \
  --url localhost:8000 \
  --request-rate 10 \
  --arrival-pattern poisson \
  --synthetic-input-tokens-mean 512 \
  --synthetic-input-tokens-stddev 128 \
  --output-tokens-mean 128 \
  --output-tokens-stddev 32 \
  --random-seed 42 \
  --request-count 200

Notes on the dynamic example:

• --arrival-pattern poisson with --request-rate 10 creates an average arrival of 10 requests/sec but with jitter, bursts and gaps, which better emulates real queuing behavior.
• Input/output stddev settings introduce prompt and output-length variance so prefill and decode costs vary per request.
• --random-seed 42 makes the sequence reproducible.
• --streaming remains required for TTFT/ITL measurements.

Under Poisson traffic, metric distributions typically widen compared to the static baseline because multiple requests compete for GPU access and prefill lengths vary. The single-concurrency case (one request at a time) gives the lowest possible TTFT but is an idealized low-throughput scenario.

Advanced use cases

AIPerf also supports multi-node Kubernetes deployments, KV cache reuse warm-up mechanics, production trace replay, prefix synthesis, custom datasets, and sweep configurations across concurrency levels. For distributed inference at scale, NVIDIA points to Dynamo 1.0 as the underpinning for multi-node inference scenarios.

Getting started and references

The AIPerf repository and documentation host the tutorials, canonical docs, and contribution guidance for new features.

Acknowledgments

AIPerf is a collaborative effort between NVIDIA and external contributors. NVIDIA thanks Loki Ravi, Dan Ferguson, and Sheng Moua (AWS) for collaboration and validation; Aaron Batilo (Coreweave) for the Weights & Biases exporter and related improvements; Shounak Ray and Michael Feil (Baseten) for Baseten trace replay support and faster trace loading; Cristian Lopez (Pinterest) for DAG benchmarking methodology; and Ben Hamm for product guidance during AIPerf’s design and implementation.