Batch Size and Latency Tradeoff

Batching is free throughput until it isn’t, the crossover from memory-bound to compute-bound tells you exactly where “free” ends.

The two regimes

$\text{TPOT}(B)=\max \left(\underset{\text{memory-bound}}{\underbrace{\frac{W+BS{k}_{\text{tok}}}{{\text{BW}}_{\text{eff}}}}},\underset{\text{compute-bound}}{\underbrace{\frac{2NB}{{\text{FLOPS}}_{\text{eff}}}}}\right)$ $\text{Throughput}(B)=\frac{B}{\text{TPOT}(B)}$ $W$ = total bytes of modal weights = params x bytes per param B = batch size (number of sequences decoding concurrently in this step) S = sequence length; tokens of context per sequence (how much KV each one has accumulated) $k_{tok}$ = KV bytes per token per sequence

$BS{k}_{tok}$ = total KV cache bytes that must be read this step Add W for the # of bytes moved per decode step.

$B{W}_{eff}$ = achieved HBM bandwidth

$N$ = param count $\to$ 2N flops per token $\to$ 2NB flops per step (assuming dense matmuls) $FLOP{S}_{eff}$ = achieved compute throughput for the relevant dtype.

• Small $B$ ($BSk\ll W$): TPOT is flat, throughput grows linearly. Batching is nearly free.
• Mid $B$: KV traffic starts to matter; throughput bends over toward its ceiling ${\text{BW}}_{\text{eff}}/(S{k}_{\text{tok}})$.
• Large $B$: compute-bound. Throughput is pinned at ${\text{FLOPS}}_{\text{eff}}/2N$ and TPOT grows linearly. Pure latency loss, zero throughput gain.

The crossover batch

A decode GEMM $[B,d]\times [d,d]$ does $2B{d}^2$ FLOPs while moving $d^2{b}_w$ weight bytes. Arithmetic intensity $=2B/b_w$. Set it equal to the device ops:byte ratio $R={\text{FLOPS}}_{\text{peak}}/{\text{BW}}_{\text{peak}}$: $B^{\ast }=\frac{R{b}_w}{2}$ The precision cancels: halving $b_w$ doubles $R$ on tensor-core hardware, so $B^{\ast }\approx 200-400\text{ tokens in flight}$ on essentially every modern accelerator, in BF16, FP8, or FP4. Count tokens in flight, not requests, chunked-prefill tokens count too.

Picking $B$ for an SLO

Invert TPOT in the memory-bound regime: $B_{\text{SLO}}=\frac{{\text{BW}}_{\text{eff}}\cdot {\text{TPOT}}_{\max }-W}{S\cdot k_{\text{tok}}}$ Example: ${\text{TPOT}}_{\max }=20$ ms, ${\text{BW}}_{\text{eff}}=2.1$ TB/s, $W=8$ GB, $S=2048$, $k_{\text{tok}}=128$ KiB: $B_{\text{SLO}}=\frac{2.1\text{e}12\cdot 0.02-8\text{e}9}{2048\cdot 1.31\text{e}5}=\frac{34\text{e}9}{2.68\text{e}8}\approx 127$ Then take $B=\min (B_{\text{SLO}},B_{\max }^{\text{HBM}},B^{\ast })$, SLO, memory, and diminishing returns.

Closing the loop with load

Little’s Law ties offered load to the batch you will actually see: $B_{\text{steady}}=\lambda \cdot \mathbb{E}[\text{latency}]$ If $\lambda \cdot \text{latency}>B_{\max }$ the queue grows without bound and P99 diverges, that is an admission-control problem, not a batching one. See Scheduling and Admission Control and Latency Decomposition.

Changes	Effect
GQA/MLA	shrinks $k_{\text{tok}}$ → raises the throughput ceiling
KV quantization	same, ~2×
Weight quantization	shrinks $W$ → flat region extends further
Speculative Decoding	trades spare FLOPs for latency; only helps below $B^{\ast }$
Disaggregated Inference	lets prefill and decode sit at different batch sizes