During autoregressive decoding, attention at each new token needs the keys and values of every previous token. Recomputing them every step would be quadratic. The KV cache stores them so each new token costs one incremental step instead. It is also the main thing that limits how many requests a server can hold at once.
What it stores
For every layer and every past token, the projected key and value vectors. Queries are not cached: only the newest token produces a query, which attends over all cached keys and values.
Size
- Factor of 2 for K and V; requests, tokens, layers, KV heads, head dim .
- It grows linearly with sequence length and batch, and can dwarf the model weights for long contexts.
Why it dominates serving
Weights are fixed, but the KV cache scales with concurrent tokens. It sets the ceiling on batch size, and reloading it each decode step is a big part of why decode is memory-bound (see Inference Metrics).
Shrinking it
- Grouped-query attention (GQA): many query heads share a few KV heads, cutting and shrinking the cache several fold with little quality loss. Multi-query attention is the extreme (one KV head).
- Quantization: store K and V in int8 or fp8.
- KV cache compression / eviction: drop or merge tokens that attention rarely attends to.
- Sliding window: keep only the most recent tokens, bounding the cache regardless of length.
Managing it
- Naive contiguous allocation wastes memory to fragmentation and over-reservation. Paged Attention fixes this with paging.
- Shared prefixes (system prompts, few-shot examples) can be cached once and reused, see Prefix Caching.