NVIDIA Dynamo is the orchestration layer that sits on top of an inference engine, working with vLLM, SGLang, and TensorRT-LLM as backends to coordinate large-scale deployments.
What it orchestrates
- KV cache re-use: retain KV Cache between requests and route on prefix match (see Cache-Aware Routing).
- Disaggregation: separate prefill and decode onto individually optimized, independently scaling engines (see Disaggregated Inference).
- Multi-node parallelism: span two or more GPU nodes in a single replica, usually with Expert Parallelism (see Model Parallelism for Inference).
Why use an orchestration layer
- Dynamo’s abstractions for distributed KV routing, disaggregation, and multi-node parallelism aggregate runtime information to make real-time configuration adjustments as traffic fluctuates.
- Example: an SLA-based planner scales prefill and decode workers up and down against user-defined TTFT and TPS targets (see Inference Metrics).
- For disaggregation specifically, it provides a prefill queue, conditional-disaggregation routing thresholds, and efficient NIXL-based KV transfer (including a kernel to transpose KV blocks when prefill and decode engines run different TP configs).