vLLM is the open-source serving engine that popularized Paged Attention and Continuous Batching.
Why it matters
By managing the KV Cache with paging and reforming the batch every step, vLLM achieves far higher throughput than naive serving at the same latency, on the same hardware.
What it brings together
- Paged attention: near-zero KV fragmentation, so many more sequences fit in memory.
- Continuous batching: iteration-level scheduling keeps the GPU full.
- Prefix caching: shared prompts reuse KV via copy-on-write blocks (see Prefix Caching).
- Tensor and pipeline parallelism: serve models larger than one GPU (see Model Parallelism, Pipeline Serving).
- Speculative decoding and quantization for extra latency and memory wins.
general Architecture
- Scheduler: decides each step which requests to prefill, which to decode, and admits or evicts sequences based on available KV blocks.
- Block manager: allocates and shares physical KV blocks, handling copy-on-write.
- Engine / worker: runs the model forward pass across the parallel GPUs and returns logits for the batch.
Mechanism-level detail (the engine loop, the unified token budget, hash-based block sharing, preemption by recompute vs swap) is in vLLM Internals.
The through-line
vLLM is best understood as the OS-style insight applied to LLM serving: treat KV memory like paged virtual memory and schedule work at the finest granularity. Almost every other note in this folder is a component it uses.
vLLM vs SGLang
Both do continuous batching, paged KV, prefix caching, and speculative decoding, so the notes overlap by necessity. The distinguishing idea is the cache data structure: vLLM hashes fixed-size blocks into a flat table, SGLang keeps a radix tree of prefixes and schedules against it. Read this note for the paging model and SGLang for prefix-aware scheduling.