Arithmetic Intensity

A GPU can be bound by compute (FLOPs/second) and memory bandwidth (bytes/second).

Ops:byte ratio

• Each GPU has a characteristic ops / byte ratio = compute speed / memory bandwidth.
• An H100 in FP16 does ~989 TFLOPS against 3.35 TB/s, giving ops:byte ≈ 295. To be perfectly balanced, the workload must do ~295 operations per byte read.

Arithmetic intensity

$\text{arithmetic intensity}=\frac{\text{total compute (ops)}}{\text{total memory traffic (bytes)}}$

Where ops:byte is a per-second hardware property, arithmetic intensity is a per-algorithm property. Compare the two on a roofline model (see Roofline Analysis):

• Compute-bound: arithmetic intensity > ops:byte → hits the flat performance ceiling.
• Memory-bound: arithmetic intensity < ops:byte → hits the diagonal bandwidth ceiling.

Why prefill and decode differ in arithmetic intensity

The most expensive operation in both phases is attention. Prefill loads the weights once and does large matmuls over the whole prompt in parallel (high arithmetic intensity), while Decode reloads all weights for every token to do a cheap matrix–vector multiply, low arithmeticintensity.

Related

Arithmetic intensity informs: batching, Quantization, Speculative Decoding, and Disaggregated Inference.