A GPU has two resources: compute (FLOPs/second) and memory bandwidth (bytes/second). A perfectly optimized system saturates both. Real systems have a bottleneck, one resource idles while the other is pinned, and finding it is the first step of any optimization. If an operation is memory-bound, no amount of compute tuning will help, and vice versa.
The inference bottlenecks
- LLM prefill (KV cache construction) is compute-bound.
- LLM decode (token generation) is memory-bound.
- Image and video generation are compute-bound.
Optimization means making the binding resource less limiting, e.g. batching makes decode less memory-bound by doing more compute per byte moved.
Ops:byte ratio
- Each GPU has a characteristic ops:byte ratio = compute speed ÷ memory bandwidth.
- An H100 in FP16 does ~989 TFLOPS against 3.35 TB/s, giving ops:byte ≈ 295. To be perfectly balanced, the workload must do ~295 operations per byte read.
Arithmetic intensity
Where ops:byte is a per-second hardware property, arithmetic intensity is a per-algorithm property. Compare the two on a roofline model (see Roofline Analysis):
- Compute-bound: arithmetic intensity > ops:byte → hits the flat performance ceiling.
- Memory-bound: arithmetic intensity < ops:byte → hits the diagonal bandwidth ceiling.
Why prefill and decode differ
The most expensive operation in both phases is attention. Prefill loads the weights once and does large matrix–matrix multiplies over the whole prompt in parallel, lots of math per byte, high intensity. Decode reloads all weights for every token to do a cheap matrix–vector multiply, few ops per byte, low intensity. A worked decode example (d=128, N=4096, standard attention) yields intensity ≈ 62, far below the H100’s 295, proving decode is memory-bound.
Practical takeaway
Computing arithmetic intensity by hand is an academic exercise, not a daily task, but doing it once builds the intuition behind nearly every serving choice: batching, Quantization, Speculative Decoding, and Disaggregated Inference all exist to move an operation across the roofline. See Inference Metrics for the latency/throughput consequences.