KV Cache Sizing

The KV Cache is the second-largest consumer of HBM after weights, and it is what actually caps concurrency.

The formula

$\text{KV bytes}=\underset{K,V}{\underbrace{2}}\cdot L\cdot n_{kv}\cdot d_{\text{head}}\cdot b\cdot S\cdot B$

Symbol	Meaning
$L$	layers
$n_{kv}$	KV heads
$d_{\text{head}}$	head dim
$b$	bytes per element (2 for FP16/BF16, 1 for FP8)
$S$, $B$	sequence length, batch

Per-token, per-sequence cost, memorize this one: $k_{\text{tok}}=2L{n}_{kv}{d}_{\text{head}}b$

Reducing $n_{kv}$

With $n_h$ query heads and group size $g=n_h/n_{kv}$, GQA divides the cache by $g$. See Attention Variants for Inference.

• MHA: $n_{kv}=n_h$
• GQA: $n_{kv}=n_h/g$, typically $g\in \{4,8\}$
• MQA: $n_{kv}=1$

MLA (latent compression)

Cache a single low-rank latent $c_t\in {\mathbb{R}}^{d_c}$ plus a small decoupled RoPE part $d_r$, shared across all heads: $k_{\text{tok}}^{\text{MLA}}=L\cdot (d_c+d_r)\cdot b$ Example $L=61$, $d_c=512$, $d_r=64$, $b=2$: $61\cdot 576\cdot 2=70$ KB/token, versus $2\cdot 61\cdot 128\cdot 128\cdot 2=4.0$ MB/token for equivalent MHA. ~57× compression. The cost is that K and V must be re-expanded (or the projections absorbed into $W_Q$/$W_O$) inside the kernel.

Worked numbers

8B model: $L=32$, $n_h=32$, $d_{\text{head}}=128$, $b=2$.

Scheme	$k_{\text{tok}}$	32K context, 1 seq
MHA ($n_{kv}=32$)	512 KiB	16 GB
GQA-8 ($n_{kv}=8$)	128 KiB	4 GB
GQA-8 + FP8 KV	64 KiB	2 GB
MQA ($n_{kv}=1$)	16 KiB	0.5 GB

How it caps batch size

$B_{\max }=\frac{\text{HBM}-W-\text{workspace}}{k_{\text{tok}}\cdot S}$ 80 GB device, $W=8$ GB, 6 GB workspace/activations/graphs → 66 GB free. With $k_{\text{tok}}=128$ KiB that is 503K cached tokens: 245 sequences at 2K context, or 61 at 8K, or 15 at 32K. Concurrency falls linearly in context length.

Fragmentation eats into this unless you use Paged Attention; reuse recovers some via Prefix Caching.

Rule of thumb

$k_{\text{tok}}\approx 2L{n}_{kv}{d}_{\text{head}}b$. Divide free HBM by it to get your total token budget, that number, not “batch size”, is what the scheduler is really allocating.