The Single-Stream Benchmark Trap: How to Evaluate AI Inference Hardware

A single-stream result measures responsiveness, not the full capacity of an inference service: the original MLPerf Inference paper separates sequential queries from server and offline workloads. We recommend testing local AI hardware against the work you intend to run, keeping response time and aggregate throughput visible together. For a buyer, the useful question is how much acceptable work the system completes within the time available, rather than which isolated token rate looks largest.

Single-stream is a valid test, with a limited scope

In MLPerf's single-stream scenario, a new query follows completion of the previous one, while its server scenario measures queries per second subject to latency requirements and its offline scenario measures batch throughput (MLPerf, section 4.5). Those scenarios answer different questions, so we would not treat a poor single-stream result as proof of poor batch capacity, or a high batch result as proof of responsive chat. Keep the sequential test when a person will wait for one answer before asking the next question. Add a concurrent test when your intended service must handle overlapping requests.

We suggest defining success before choosing the hardware: a response-time limit for interactive work, or an acceptable completion window for a background batch. We recommend writing those requirements down before the first run and keeping them unchanged while comparing configurations. Write the requirement in application terms, such as accepted documents completed before the next working day, without assuming a particular model will meet it.

Batching can improve throughput, but concurrency is not free

For Transformer serving, the PagedAttention paper explains that batching amortises model-weight movement across requests. That is a reason to measure concurrent throughput, not a guarantee that each additional request adds the same capacity. The same paper describes compute overhead becoming dominant at sufficiently large batch sizes, and request KV caches constraining the batch size. It also distinguishes parallel prompt processing from sequential token generation, so a short-input test does not establish performance on long inputs.

The serving scheduler matters too: ORCA introduced iteration-level scheduling so finished requests can leave and newly arrived requests can be considered between generation iterations. Submitting many requests is therefore not, by itself, evidence that all of them execute together. We recommend recording both client-side outstanding requests and the serving system's running and waiting requests wherever those measurements are available. Avoid promising unchanged per-user speed as concurrency rises, and measure the latency distribution instead.

Design the test around the application

For a document extraction workload, we recommend a fixed evaluation set with representative input lengths and explicit checks for required fields. Record accepted documents per unit of elapsed time alongside generated tokens, and include rejected outputs and retries in the run report. Treat that as an evaluation method, not a claim that concurrency improves extraction accuracy.

For a shared chat service, test the anticipated arrival pattern rather than only releasing every request at once. Report time to first token, completion time and the slower end of the response-time distribution. The MLPerf paper explicitly treats arrival patterns and tail-latency constraints as parts of realistic inference evaluation. We would keep a high-throughput configuration out of an interactive deployment if it failed the response-time requirement agreed for that deployment.

For an agent workflow, benchmark independent model calls separately from steps that require an earlier result. Include retrieval, tool execution and final synthesis in a separate end-to-end measurement. Do not present the model server's token rate as the measured duration of the whole workflow.

Run a controlled concurrency sweep

We suggest starting with one outstanding request, then increasing the limit in stages until you cover expected demand and the failure boundary you are authorised to test. Keep the input set, output limits, model weights and quality checks fixed during that sweep. Record the exact serving software, quantisation and configuration rather than reporting only a hardware name. Repeat runs, retain their individual results and identify whether model loading and warm-up are included in elapsed time.

For each stage, publish aggregate output tokens per second, accepted application results per second, request latency and failure counts. State whether token throughput covers the complete run or only a selected generation interval. If you change the model, run a separate quality comparison rather than attributing every difference to concurrency. If requests time out, preserve those failures in the result instead of calculating throughput from successes alone.

We recommend choosing a working limit that meets the application requirement, rather than automatically selecting the concurrency with the largest token total. Then test it again with the input-length mix and competing work expected in service. Present any unused capacity as an observed margin under those conditions, not as a guarantee for every future workload.

Make the purchase decision from the evidence

Our proposed buying test is a reproducible workload report with a clear pass condition, a latency distribution and an output-quality result. Ask the supplier to retain the configuration and raw measurements so another engineer can repeat the comparison. Keep single-stream responsiveness in that report where it matters, but do not let it stand in for every other operating condition. For help defining an evaluation before committing to hardware, let's have a chat.

Update, October 2026

Our later DGX Spark concurrency benchmark provides a hardware-specific concurrency sweep, while the separate pipeline article discusses a model and runner change. Read these as distinct comparisons, not as a controlled experiment proving that concurrency alone caused every reported improvement.

Sources

• Reddi and colleagues, "MLPerf Inference Benchmark", version 1, 6 November 2019.
• Kwon and colleagues, "Efficient Memory Management for Large Language Model Serving with PagedAttention", version 1, 12 September 2023.
• Yu and colleagues, "ORCA: A Distributed Serving System for Transformer-Based Generative Models", July 2022.