Prefix Caching only pays off if the request lands on a replica that actually holds the matching KV Cache.
The idea
Route a request to the replica most likely to already have its prefix cached. A user in a long chat or asking repeated questions about a codebase should keep hitting the same replica so each turn gets a fast, cheap cache hit instead of a fresh prefill.
Where to store the KV cache
The KV cache is valuable but VRAM is scarce, you configure what fraction of free VRAM the engine allocates to it, and it fills fast. Once full, blocks are evicted and future requests risk cache misses. To buy room, offload down a memory hierarchy (fastest to slowest):
| Level | Memory | Speed | Size |
|---|---|---|---|
| G1 | GPU VRAM | TB/s | 10s–100s GB |
| G2 | CPU RAM | 10s–100s GB/s | 100s GB–TB |
| G3 | Local SSD | 5–10 GB/s | TB |
| G4 | Networked SSD | GB/s | 10s TB |
- Keep hot blocks in high-bandwidth memory; relegate cold blocks to slower tiers. SKUs like GB200 have fast G2 links, making them great for offloading.
- NVIDIA Dynamo’s KVBM (KV Block Manager) provides APIs for moving blocks between levels (see NVIDIA Dynamo).
Routing across replicas
- Route on prefix match, not just replica load: a hot G1 cache beats reading the same sequence from G4.
- A global KV cache over G4 networked storage lets any replica eventually reach any precomputed sequence, so cached data survives nodes cycling or scaling down during autoscaling.
- Related routing signals include LoRA-aware routing (send a request to a replica already holding its fine-tuned weights).
It's a system property
Cache-aware routing is where the KV Cache, Prefix Caching, autoscaling, and the request router all meet. Orchestrators like NVIDIA Dynamo make prefix-aware routing decisions across a fleet. For contexts too large even to cache efficiently, see Solving Long Context Inference.