Serving a trained model well is a different problem from training it. Inference adds latency as a first-class concern, is dominated by memory bandwidth rather than compute, and lives or dies on how the KV Cache is managed.
Fundamentals
- Inference Metrics: TTFT, TPOT, throughput, and the latency / throughput trade-off.
- Arithmetic Intensity: the ops:byte ratio and roofline that decide compute- vs memory-bound.
- KV Cache: what it stores and why decode is memory-bound.
- Disaggregated Inference: split the compute-bound prefill from the memory-bound decode.
Throughput techniques
- Continuous Batching: refill the batch at every step instead of waiting.
- Paged Attention: virtual memory for the KV cache, near-zero waste.
- Prefix Caching: reuse the KV of shared prompts.
- Quantization: lower precision for more FLOPS and bandwidth — the one lossy technique.
Latency techniques
- Speculative Decoding: a draft model proposes, the target verifies in parallel.
- Attention Optimization: FlashAttention, and sub-quadratic attention variants.
- FlashAttention: the IO-aware exact attention kernel.
- Pipeline Serving: split a model across GPUs when tensor parallel stops scaling.
Memory & caching
- Cache-Aware Routing: route to the replica holding the prefix; tier the KV cache across VRAM/RAM/SSD.
Parallelism & scaling
- Model Parallelism for Inference: tensor parallel for latency, expert parallel for throughput, multi-node.
Kernels & tooling
- Kernel Fusion: cut HBM round-trips by fusing memory-bound ops.
Inference engines / orchestration engines
- vLLM: the reference engine tying these ideas together — broadest support.
- SGLang: community engine strong on large MoE models and diffusion.
- NVIDIA Dynamo: orchestration layer over the engines for large-scale, disaggregated serving.
Modalities
- Image and Video Generation Inference: compute-bound iterative denoisers, a different world.
Production
- Autoscaling: match replicas to demand without missing SLAs or wasting GPUs.