Disaggregated Inference

Prefill and decode have opposite hardware profiles, yet a single server runs both and they interfere. Disaggregated serving splits them onto separate pools of GPUs so each can be tuned and scaled independently.

The mismatch

• Prefill is compute-bound: the model processes the whole model in a single forward pass to build the initial KV cache. The phase processes the entire input prompt in parallel to initialize the KV cache, and its dominated by dense matmuls (of (seq_len, hidden) activation matrices against weight matrices — GEMMs), making it compute bound.
• Decode is memory-bound: it generates tokens one at a time autorgressively, repeatedly reading and writing the kv cache, so it is memory-bandwidth-bound. Each step is. GEMV (matrix-vector multiply) per request.

When run together, a long prefill stalls everyone’s decode and inflates inter-token latency (see Inference Metrics).

How it works

1. Prefill workers process the prompt and produce the KV Cache.
2. The KV cache is migrated over fast interconnect to a decode worker.
3. Decode workers generate tokens, running large continuous batches of decode-only steps.

Payoff

• Each pool is sized and batched for its own bottleneck, improving goodput under latency targets.
• Prefill spikes no longer disturb steady-state decode, so TTFT and TPOT are both more predictable.
• Pools scale independently: add prefill capacity for prompt-heavy traffic, decode capacity for long generations.

The cost

Moving the KV cache between pools consumes interconnect bandwidth and adds latency, so disaggregation pays off mainly at scale. Below that, chunked prefill (interleaving prompt chunks with decode on one server) captures much of the benefit without the transfer.

KV Migration mechanisms

• What actually gets transferred is the key and value tensors of shape [num_layers, num_heads, seqa_len, head_dim]
• In vLLM, it’s implemented as a KV connector abstraction.
  • Disaggregated servings separates the prefill/decode phases onto different vLLM isntances; a prefiller computes the KV cache blocks, then transfers those blocks over the network to a decoder that proceeds directly to token generation.
  • NIXLConnector does RDMA-styl
    • You can also have layer-wise / overlapped transfer where the system can stream KV cache layer by layer as each transformer layer finishes computing during prefill.
      • In NIXL, you can move KV cache directly from prefill-engine VRAM to decode-engine VRAM via RDMA rather than staging through host memory.
• Example implementations:
  • https://github.com/ai-dynamo/nixl
  • https://github.com/kvcache-ai/Mooncake

Control Plane

• An orchestration layer is needed to:
  • Route incoming requests to a prefill worker (based on load, KV-cache prefix reuse potential, etc)
  • Track completion of prefill + trigger the KV transfer
  • Assign a decode worker and hand off the request + transferred KV cache to it
  • Manage two independent quesue
• Commonly used systems are:
  • NVIDIA Dynamo: requests arrive at the decode worker first, which decides via its internal router.