Many requests share the same opening tokens: a long system prompt, a few-shot template, a common document. Prefix caching computes the KV Cache for that shared prefix once and reuses it across every request that starts the same way, skipping redundant prefill.
The idea
The KV of a token depends only on the tokens before it. So the KV for a shared prefix is identical across requests and can be computed once, stored, and reused.
How it works
- Hash the prefix (often block by block) and key the cached KV blocks by that hash.
- A new request whose prefix matches reuses those blocks and only prefills the tokens after the shared part.
- Built on Paged Attention: shared blocks are pointed to by multiple sequences via copy-on-write, so no data is duplicated.
Payoff
- Cuts prefill compute and time to first token (see Inference Metrics) for workloads with heavy prompt reuse.
- Biggest wins: shared system prompts, few-shot prompts, multi-turn chat (each turn reuses the whole prior conversation), and RAG with repeated context.
Limits
Only an exact prefix match helps: a single differing earlier token invalidates everything after it. Cached blocks also consume memory, so they are evicted under pressure, usually least-recently-used.