Serving is judged on latency and throughput, and the two trade off against each other. Knowing which metric a workload cares about decides every serving choice, from batch size to whether to use Disaggregated Inference.
The two phases
- Prefill: process the whole prompt in one pass. Compute-bound, sets the time to first token.
- Decode: emit tokens one at a time, each a forward pass over one token. Memory-bound, sets the time between tokens.
The metrics
- TTFT (time to first token): how long until the first output appears. Dominated by prefill and by queueing delay. Matters most for interactive and streaming use.
- TPOT / ITL (time per output token, inter-token latency): the steady-state gap between tokens once generation starts. Set by decode speed.
- End-to-end latency: .
- Throughput: total tokens per second across all concurrent requests. The metric for batch and offline jobs.
- Goodput: throughput that also meets a latency target (SLO), the number that actually matters in production.
The trade-off
- Larger batches raise throughput (better arithmetic intensity, see Roofline Analysis) but raise per-request latency.
- Smaller batches cut latency but leave the accelerator underused.
- Continuous Batching pushes this frontier out by keeping the batch full without making any one request wait for a fixed window.
Why decode is memory-bound
To generate one token, the GPU reloads every weight and the entire KV Cache from HBM, then does a tiny amount of math. Throughput is set by memory bandwidth, not peak FLOPs. Batching many requests amortizes the weight reads across many tokens and is the single biggest throughput lever.
Match the metric to the workload
Chatbots optimize TTFT and TPOT. Bulk data processing optimizes throughput. Serving both from one pool usually means separating prefill and decode so a long prompt does not stall everyone’s decode.