NVIDIA’s successor to GenAI-Perf separates load generation from result processing and lets teams replay varied traffic with repeatable settings.

When a company runs an AI model, “How fast is it?” has no single answer. A system may respond quickly to one short prompt but slow down when many users arrive with long prompts at once.

NVIDIA’s AIPerf is designed to measure those differences. It is a ground-up successor to GenAI-Perf, NVIDIA says, with a new architecture for creating workloads, collecting results and reporting several kinds of latency and throughput.

The practical goal is simple: make sure the benchmarking tool does not become the slowest part of the test.

Keeping the test tool out of the way

A benchmark sends requests to an AI server, then records what happens. A simple script might do both jobs in one process. That can fail under heavy traffic: the script itself may not create requests or process responses quickly enough.

AIPerf uses multiple worker processes to generate requests. Separate record-processing services handle the results, while ZeroMQ, a messaging system for connecting processes, coordinates the parts. NVIDIA says this multiprocess design helps prevent AIPerf from becoming a client-side bottleneck.

That matters because a weak load generator can make a powerful model server look slower than it is. It can also make comparisons unreliable if the tool struggles differently at different request rates.

AIPerf is not promising that every benchmark becomes automatically accurate. The hardware, model server, network and workload still shape the result. Its design addresses one specific source of distortion: the benchmark client running out of capacity.

Testing more than one kind of traffic

A useful benchmark must resemble the service being measured. Real users do not all send prompts of the same length at perfectly even intervals.

NVIDIA says AIPerf supports more than 15 endpoint types, including chat, responses, NIM rankings and image generation. It also supports public datasets such as ShareGPT and trace-replay formats from Mooncake, Baseten and WEKA’s AgentX.

That lets an engineer choose between a quick synthetic test and a replay of captured traffic. Synthetic traffic is generated by the tool rather than taken from users. Trace replay attempts to reproduce a recorded pattern of requests.

The tool also lets users control how requests arrive. Available patterns include constant, Poisson and gamma arrivals. A Poisson pattern means requests arrive with random gaps around an average rate, rather than on a perfectly fixed schedule. Gamma arrivals can provide different levels of burstiness.

This distinction is important. A constant stream might be useful for a clean baseline, but it can hide queueing problems caused by bursts. A variable workload can show how the server behaves when requests overlap and compete for the same accelerator.

Why fixed and variable tests belong together

NVIDIA’s walkthrough uses Qwen3-0.6B served through vLLM. The small model is meant to make repeated experiments quick; NVIDIA says the example establishes a measurement process rather than serving as a definitive test of that model.

The first example holds each request to exactly 128 input tokens and 128 output tokens. Fixed lengths make it easier to compare one server configuration with another because the work stays consistent.

The second example loosens those constraints. It averages 10 requests per second across 200 requests, uses an average input length of 512 tokens with a standard deviation of 128, and varies output length around 128 tokens with a standard deviation of 32.

It also uses a random seed of 42. A random seed initializes the workload’s random choices, allowing the same sequence of request timings and token lengths to be recreated.

That reproducibility is useful when an engineer changes one setting, such as the model server or hardware. Running the same workload again reduces the chance that a different random traffic pattern explains the result.

The dynamic example uses Poisson arrivals, so requests cluster and spread out instead of arriving at a perfectly steady pace. NVIDIA reports that this produces wider latency distributions than the fixed test. That is an expected result of mixing prompt lengths and overlapping requests, not necessarily a sign that the server has become less efficient.

Four numbers answer different questions

AIPerf reports several measurements rather than one universal speed score.

Time to first token, or TTFT, measures how long a request takes to produce its first piece of text. This is especially important for interactive applications, where users notice the initial pause.

Inter-token latency, or ITL, measures the time between successive tokens during generation. A system can start quickly but produce the rest of an answer slowly, so TTFT alone can hide that problem.

Request latency measures the full time from request to completed response. It combines the initial prompt-processing phase with token generation.

Output-token throughput measures how many output tokens the system generates per second across concurrent requests. This is useful for thinking about capacity: how much work can the service handle at once?

AIPerf reports these metrics with percentiles such as p50, p90, p95 and p99, along with averages, minimums, maximums and standard deviations. A percentile shows how a particular share of requests performed. For example, p99 describes the point below which 99 percent of observations fall.

Percentiles expose the slow tail. A service can have an acceptable average TTFT while a small but important group of users waits much longer. That difference may matter more than the average for a customer-facing application.

What engineers should do with it

AIPerf’s main contribution is structure. It separates the work of producing traffic from processing measurements, supports several workload shapes and records metrics that describe both responsiveness and capacity.

Teams should begin with a fixed workload for clean comparisons, then test variable prompt lengths and arrival patterns that resemble their service. They should compare percentiles, not just averages, and use a fixed random seed when repeating experiments.

NVIDIA’s documentation demonstrates these workflows and describes AIPerf’s capabilities. It does not, by itself, establish how much its measurements differ from GenAI-Perf or other load generators across hardware, model servers and production systems. That comparison is the next practical test for teams deciding whether to adopt it.