Below is a complete article built from the provided post content, with a clear structure, an FAQ section, and a conclusion section added.
—
## Beyond Raw Throughput: Why Goodput Matters for LLM Serving
When we benchmark an LLM serving setup, the number almost everyone reaches for first is throughput: how many requests per second the system can push through. It is easy to measure, easy to compare, and it lines up nicely with the thing finance actually cares about, which is dollars per request. So we chase it, tuning batching, raising concurrency, squeezing the GPU, and watching requests per second climb.
The trouble is that this number, on its own, can keep rising while your service quietly gets worse. I ran into exactly that while tuning vLLM on a single GPU and the gap was wide enough to be worth walking through because the lesson travels far beyond my particular setup.
### Throughput Counts Plates, Goodput Counts Happy Diners
Picture an LLM endpoint as a restaurant kitchen. Throughput is the number of plates that leave the pass every hour. It tells you the kitchen is busy but it says nothing about whether the food arrived hot, on time, or at the right table. A kitchen can double its plates per hour and still send a full room of cold and late dinners.
The metric that captures what we actually want is goodput. The definition is refreshingly plain: it is the number of completed requests per second that also meet your latency targets. For LLM serving those targets are usually time to first token (TTFT), the wait before anything appears on screen, and time per output token (TPOT), the pace at which the rest of the answer streams out. A request that finishes but blows past either target is a plate that left the kitchen cold. It still counts toward throughput but it does not count toward goodput.
### The Setup, and How I Drove Load
Everything here runs on parts you probably already have. I served Qwen2.5-7B with vLLM on a single NVIDIA A10G with 24 GB of memory inside an EKS cluster, drove load with GuideLLM, and watched the system through Prometheus and Grafana with GPU-level metrics coming from NVIDIA’s DCGM exporter. The entire setup is on GitHub, Kubernetes manifests and all, so you can reproduce my runs and, more usefully, aim the same benchmark at your own model and real traffic instead of the synthetic load I used.
One honest note before the numbers. A single A10G is not the GPU you would pick for a production-grade LLM service, so the absolute numbers here will look modest next to an H100 cluster. I picked it on purpose to keep these studies cheap enough to repeat often. The point of this post is not the ceiling, it is the shape of the trade-off between throughput and latency, and that shape holds regardless of how big the GPU is.
I focused on changing only three vLLM settings:
– `gpu_memory_utilization`
– the batched token budget (`max_num_batched_tokens`)
– the cap on concurrent sequences (`max_num_seqs`)
I picked these three on purpose because together they decide the throughput against latency trade-off. They control how much GPU memory is left for the KV cache, how much work goes into each batch step, and how many requests run at the same time. They are also the settings an operator can safely change in production without touching the model, the hardware, or the cluster, so holding everything else fixed keeps the results easy to read and easy to reproduce.
How you send the traffic matters just as much as how you tune it, so it is worth being explicit about it. I used GuideLLM in two ways:
– A rate sweep ramps the request rate up gradually and lets me read latencies while the server still keeps up, so the numbers reflect real serving and not a backlog.
– A throughput run does the opposite, firing requests as fast as the server will take them to find the ceiling, which pushes it into saturation.
I will use both in this post, and the contrast between them is part of the lesson.
### What the Studies Show
I ran the same setup against three kinds of traffic because the shape of the traffic decides which latency target actually bites:
– Chatbot: ~485 / 121 tokens, tight TTFT, medium TPOT
– Reasoning: ~400 / 4000 tokens, TPOT, with TTFT loose
– Agentic: ~512 / 128 tokens, per-call TTFT that adds up
#### Chatbot Study: Constrained Goodput in Action
The chatbot study is the clearest case. The goal was to maximize combined token throughput (prefill_token_throughput + decode_token_throughput) under one constraint: average TTFT had to stay at or below 1.5 seconds. I also added a windowing rule, scoring each config from a stretch of eight consecutive data points where prefill throughput had settled down rather than from any single sample. In a live service, jitter smooths out only over steady windows, so this rule measures real throughput instead of a blip.
You can see that jitter and the windowing rule doing its job in the raw metrics from two configs I will use as our main example. Both configs ramp up over the run as GuideLLM raises the load, both wobble step to step, and both still let you read off a believable plateau once the ramp settles.
| Config | gpu_memory_utilization | max_num_batched_tokens | max_num_seqs | TTFT avg | Prefill tput | Decode tput |
|——–|————————|————————|————–|———-|————–|————-|
| Exp 1 | 0.89 | 4160 | 212 | 1395 ms | 1925 tok/s | 478 tok/s |
| Exp 20 | 0.87 | 2150 | 271 | 1403 ms | 2934 tok/s | 690 tok/s |
Both configs land at almost the same TTFT, right under the 1.5 second ceiling. That is the constraint doing exactly what I asked it to do: hold the first-token wait at the edge of what I said was acceptable and then squeeze as much combined throughput out of the rest of the system as it can. Exp 20 wins that squeeze by a wide margin, with about 50 percent more combined throughput than Exp 1.
That extra throughput is not free, and the bill shows up somewhere I did not constrain. Exp 1’s p95 TPOT sits at about 50 milliseconds. Exp 20’s sits at about 494 milliseconds, nearly ten times higher. I told the search to protect the first token. I never told it to protect the pace of the stream that follows, so it spent that budget without asking.
The two latencies move in different directions because they come from two stages of the same request:
– TTFT comes from the prefill stage (reading the prompt).
– TPOT comes from the decode stage (generating subsequent tokens).
Pushing more combined throughput means packing more work into each decode step, and a fuller step takes longer. Picture the kitchen sending one course to every table each round: a fuller room makes each round longer and every table waits more between courses. Constrain how fast the kitchen seats you and brings the first course, and the wait gets pushed onto the gap between the courses that follow.
#### Reasoning and Agentic Studies: No Constraint, Different Questions
Reasoning and agentic ran a different experiment on purpose: I kept the same goal (maximizing combined prefill and decode throughput) but with no TTFT constraint, and I drove load with a throughput run, sending requests as fast as the server would take them to find the ceiling.
– In the agentic study, two configs were nearly tied on requests per second (6.20 vs 6.11), yet their p95 TPOT differed by about 220 milliseconds (682 vs 902). Same throughput, noticeably different stream quality.
– In the reasoning study, the config with the best raw throughput was not the one you would want once you cared about latency: tightening the inter-token target pushed the winner to a different configuration entirely.
Three different studies, three different ways of asking the question, and the same answer comes back each time: a throughput number with nothing else attached does not tell you which configuration to run.
### Optimize for the Thing You Actually Sell
Three practical lessons for anyone running inference in production:
1. A throughput number with no SLO next to it is marketing, not engineering. The honest target is goodput—requests per second that actually meet your latency targets. Count anything else and you are counting cold plates.
2. The config with the best goodput is not the one you would guess. The space is full of small cliffs where nearby settings behave very differently. The reliable way through it is an automated, data-driven search that tries many configs against your real targets rather than a few hand-picked guesses.
3. None of this stays put. Traffic, prompts, and models all change over time, and the best config moves with them, so the search has to run again, not just once.
There is also a broader point: I changed only three serving knobs and the gap was already this wide. The same trade-off lives in the layers I held fixed, in the GPU settings, the runtime, and the scheduling. The largest gains often come from tuning the whole stack together since the layers interact.
### Practical Takeaway
The practical move is small and worth doing early:
– Set your TTFT and TPOT targets first.
– Then look for the config that gives you the most throughput while still meeting them.
– You will often ship a smaller requests-per-second number and a clearly better service—the trade your users would make every time.
If you want to see the gap on your own hardware, the full setup is on GitHub at [github.com/graz-dev/vllm-benchmark](https://github.com/graz-dev/vllm-benchmark). Clone it, point GuideLLM at your own model and traffic, set your SLOs, and search for the best goodput rather than the best raw throughput. That gap is usually where your real headroom is hiding.
—
## FAQ
**What is the difference between throughput and goodput in LLM serving?**
Throughput counts all completed requests per second. Goodput counts only those requests that meet your latency targets for time to first token (TTFT) and time per output token (TPOT).
**Why does optimizing only for throughput lead to worse user experience?**
Because throughput can rise while latency worsens. Users feel stuttering streams when inter-token times are high, even if overall requests per second look good.
**What are TTFT and TPOT, and why do they matter?**
TTFT is the time until the first token appears (perceived initial responsiveness). TPOT is the pace at which subsequent tokens arrive (perceived smoothness). Both together define the user experience.
**Which settings should I tune to improve goodput?**
The most impactful serving knobs are typically `gpu_memory_utilization`, `max_num_batched_tokens`, and `max_num_seqs`. These control batching, concurrency, and KV cache memory.
**Should I always maximize throughput?**
Not if you care about user experience. You should maximize goodput: throughput constrained by your latency SLOs. The best raw throughput is usually not the best production configuration.
**How do I find the best configuration for my workload?**
Use an automated, data-driven search that evaluates many configurations against your real traffic patterns and latency targets, rather than hand-tuning a few options.
**Does the best configuration stay the same over time?**
No. As traffic, prompts, and models evolve, the best configuration changes, so continuous benchmarking is necessary.
**Is one GPU benchmark enough for production?**
A single-GPU benchmark is useful for understanding trade-offs and finding headroom, but production-scale testing on your target hardware and traffic patterns is still essential.
—
## Conclusion
Optimizing LLM serving is not about pushing raw throughput numbers as high as possible. It’s about maximizing goodput—completed requests per second that satisfy your latency targets. A high throughput number with poor latency is cold food on a slow tray. By measuring goodput, tuning with real targets, and continuously re-evaluating as workloads change, teams can unlock real headroom, deliver smoother user experiences, and make infrastructure choices that truly align with what they are selling.
