Insights27 March 2026Updated 26 October 20265 min read

Reasoning Effort: Test the Token Budget Before Buying More Capacity

Cyan and magenta beams of light fanning out from a small glowing square on a black background

OpenAI describes gpt-oss as offering low, medium and high reasoning effort for different use cases and latency needs. We recommend testing reasoning effort before changing an extraction pipeline's hardware. Our recommendation is to compare those settings against accepted output, rather than assume the lowest effort is always the fastest useful configuration.

In vLLM's gpt-oss integration, reasoning effort becomes part of the model's system message. That effort instruction is not a hard cap restricting reasoning to a handful of tokens.

Separate effort from the output budget

Two glowing gauges side by side, one filled with swirling particles, the other with a short neat bar

The vLLM v0.11.0 Chat Completions request schema accepts a top-level reasoning_effort field with low, medium or high values. Its Harmony integration maps that value to the corresponding reasoning-effort instruction, rather than assigning a separate numeric reasoning allowance. Those are the mechanics of this version's integration, not a promise about every server using an OpenAI-compatible API.

The same version's output-budget calculation uses max_completion_tokens or max_tokens, constrained by other applicable limits. The scheduler stops a request when its generated output reaches the token allowance or the sequence reaches the model-length limit. For gpt-oss, the server then separates reasoning and final content from the generated token sequence. Taken together, the generation stop check and response parsing show that the allowance is not reserved exclusively for the final JSON object.

We recommend checking this contract on the actual model, server version and endpoint before copying a request configuration. Record the output-limit field you send, the effort setting, the model revision and the serving software version together. Keep that record with the evaluation results so the next configuration change has a reproducible baseline.

Check how the response finished

Several streams of light, most ending in a clean bright cap, one cut off abruptly at a hard edge

A response ending at a generation limit has a different finish reason from one that stopped normally. In this vLLM release, a length-capped request maps to the length finish reason, which the engine describes as consuming max_tokens or reaching max_model_len. The Chat Completions handler carries the finish reason into the response alongside the message.

We recommend inspecting that finish reason before diagnosing malformed JSON as a model-quality problem. Count a length-limited response separately from a transport error, a parsing failure and a record rejected by business validation. Retain enough metadata to distinguish these outcomes without collecting confidential source text in routine logs.

In vLLM v0.11.0, the gpt-oss Chat Completions handler returns reasoning in reasoning_content and final text in content. We recommend verifying the response shape against your deployed version instead of assuming the reasoning field has the same name across providers. Treat an absent final answer as a failed extraction, even if the request returned a response object.

Test low effort without assuming equal quality

We recommend including low effort in the evaluation for extraction tasks where the required fields are explicit in the input. Use the same records and acceptance rules for the comparison with medium or high effort. Include difficult records, missing values and inputs that previously caused incomplete answers, rather than testing only the shortest examples.

Keep the model, prompt, output allowance and concurrency unchanged while comparing effort settings. Review factual field accuracy as well as whether the answer parses. Treat a syntactically valid object with the wrong values as a failed result, not as evidence that the configuration works.

For tasks requiring inference beyond the supplied text, we recommend assessing that reasoning requirement explicitly rather than grouping them with direct extraction. Choose the effort setting from the quality and completion results, not from the task's label alone. Do not infer that low effort disables reasoning, guarantees short reasoning or preserves quality on every workload.

Measure records the application can use

We recommend defining throughput as accepted records over elapsed time for the evaluation, with acceptance rules written down before it starts. Count retries and rejected records in the run's resource use, rather than reporting only the successful requests. Keep request latency visible alongside the aggregate rate so a faster batch does not conceal unacceptable waits for individual records.

In chapter 8 of Modern Software Engineering, David Farley describes why average latency missed outliers that mattered to a trading system. Our recommendation is to apply that distinction to extraction: retain the slow and failed cases when comparing settings, rather than reducing the run to one average.

Record the input set, acceptance checks, token allowance, effort setting and full run duration with each result. Compare repeated runs before describing a setting change as a capacity improvement. Require evidence of both usable output and acceptable latency before deciding whether to keep the change.

Make the configuration decision reversible

We recommend changing reasoning effort separately from concurrency, model size and hardware so the comparison answers a specific question. If the lower-effort run fails the agreed quality checks, restore the previous setting and investigate those records before widening the rollout. If output truncation persists, review the generation allowance and response requirements rather than treating low effort as a guaranteed fix.

Keep the chosen configuration and its evaluation records together, including the cases it did not handle well. Re-run that evaluation after changing the model, prompt or serving implementation. Use the accepted-record rate and the failed cases to decide the next change, whether that is a larger output allowance, a different effort setting or more capacity.

Update, October 2026

For a separate discussion of concurrent inference, see our later DGX Spark concurrency benchmark. We recommend treating hardware and concurrency results as a separate experiment from a reasoning-effort comparison, not as proof of the effect of this setting.

Sources

Want this in your team?

Tell us what you're building and where you want to get to, and we'll think it through with you.

Let's have a chat Follow by RSS →

All insights →