SGLang

An open-source LLM serving engine built around RadixAttention: the KV cache is stored as a radix tree over token prefixes, so requests can automatically reuse KV from previous requests with the same prefix.

The other big ideas are cache-aware scheduling and hiding scheduler overhead behind GPU execution. vLLM and TensorRT-LLM are the main comparisons.

Process architecture

Four roles, connected with ZMQ over ipc://:

Component	Role
HTTP + TokenizerManager	Tokenization, request IDs, per-request state
Scheduler	Waiting queue, radix tree, KV memory management
TpModelWorker / ModelRunner	Forward pass + sampling
DetokenizerManager	Streaming detokenization

There’s one scheduler per TP rank, but only rank 0 talks to ZMQ. Rank 0 broadcasts requests to the other ranks through a Gloo CPU group.

HTTP → tokenize → scheduler → GPU forward
                              ↓
HTTP ← detokenize ← results ←

The processes are separated so CPU work can overlap with GPU work.

RadixAttention

The KV cache is a compressed trie over token sequences.

Each node stores:

• a run of token IDs

• KV-pool indices for those tokens

• child nodes

• lock_ref for active requests using the node

class Node:
    def __init__(self, key=(), value=None):
        self.children = {}
        self.key = key
        self.value = value
        self.lock_ref = 0

match_prefix() walks the tree and returns the cached KV for the longest matching prefix. If a request diverges in the middle of a node, the node is split.

Insertion works the same way. Duplicate KV can be freed when the tree already contains the same prefix.

Eviction

Only unlocked nodes can be evicted. Eviction is leaf-first + LRU.

Removing a leaf can turn its parent into a leaf, making the parent the next eviction candidate. This naturally keeps shared prefixes alive.

There isn’t a separate cache-only memory region:

available KV = free KV + evictable KV

Why a radix tree?

Hashed block caching only shares prefixes at block boundaries. With a radix tree and page size 1, reuse can continue all the way to the exact token where requests diverge.

The downside is that tree operations happen on the CPU critical path.

Larger page sizes also reduce hit rate because partial pages aren’t matched or inserted.

Cache-aware scheduling

Because the tree tells you how much of each request is cached, the scheduler can use that information when building batches.

• LPM: prioritize requests with the longest cached prefix.

• DFS-weight: prioritize branches shared by the most queued requests.

• In-batch prefix caching: if several requests share a cold prefix, run one first so the others can reuse its KV.

This costs CPU time, though, so implementations can fall back to FCFS when the queue gets large.

Zero-overhead scheduler

The scheduler tries to stay one batch ahead:

plan = get_next_batch_to_run()
result = run_batch(plan)          # GPU: batch N
process_results(previous)         # CPU: batch N-1
previous = result

The key trick is the future map: decode input IDs come from the previous iteration’s sampled-token tensor, which is still on the GPU, so there’s no CPU sync every iteration.

Two exceptions:

• Consecutive prefill batches aren’t overlapped to protect TTFT.

• Grammar-constrained decoding needs sampling after the previous batch is processed because grammar state depends on the sampled token.

KV memory

There are two main levels of indirection:

req_to_token[request, position]
          ↓
     KV pool index
          ↓
     actual KV tensors

req_to_token is basically the block table from PagedAttention, but token-granular by default.

For MHA, KV is stored roughly as:

(size, heads, head_dim)

For MLA:

(size, 1, kv_lora_rank + rope_dim)

The allocator tracks free pages, with the free-page state itself stored on the GPU.

Retraction

If decode runs out of KV space, requests can be retracted:

1. remove them from the decode batch

2. free their KV immediately

3. requeue them

4. re-prefill them later

The freed KV isn’t inserted into the radix tree because the memory is needed immediately.

Constrained decoding

Grammar constraints maintain a state and produce a vocabulary mask.

The interesting optimization is jump-forward decoding. If the grammar says only one continuation is possible for several tokens, those tokens can be generated as a single prefill instead of token-by-token decoding.

The request is re-enqueued, and the radix tree lets it reuse its existing KV.

Parallelism

• TP: one scheduler/tree per rank; trees stay identical.

• DP attention: useful for MLA since it avoids replicating the single KV head across TP ranks. Idle ranks run synthetic batches to stay synchronized.

• EP: experts are sharded across ranks with all-to-all dispatch/combine and load balancing.

• PD disaggregation: prefill and decode run separately, with KV transferred over RDMA.