The three forms
| Method | Mechanism | Best for |
|---|---|---|
| Pipeline (PP) | splits the model across layer depth | multi-node only; poor single-node latency |
| Tensor (TP) | splits tensors within each layer | low latency inside a node |
| Expert (EP) | shards whole MoE experts across GPUs | high throughput on MoE models |
This is the inference-side view of the parallelism families in Model Parallelism, Pipeline Parallelism, and Expert Parallelism. topology-aware parallelism is when you design around the interconnect constraints.
Tensor Parallelism, for latency
- The default for multi-GPU inference; works for dense (Llama 405B) and MoE models.
- Each layer is split across GPUs so the cost of reading weights and doing matmuls is shared, but every layer needs an all-reduce to combine results before the next layer.
- That communication is only cheap over high-bandwidth intra-node NVLink/NVSwitch, which is why TP is a poor fit across nodes. When the model is large enough, higher TP improves per-user TPS (see Inference Metrics).
Expert Parallelism, for throughput
- Divides MoE experts across GPUs (128 experts in EP8 → 16 experts/GPU). Each token still takes the same time, but the system handles more tokens at once.
- Communication is lower than TP: the small expert router is replicated on every GPU, and GPUs only pass tokens expert-to-expert: no per-layer result collection. So EP scales well to multi-node and slower interconnects.
- Many deployments mix TP and EP: TP for the attention layers, EP for the sparse MoE layer.
Multi-node inference
- Needed for huge models, multi-million-token contexts, or maximum speed: but adds InfiniBand (much slower than NVLink) between nodes.
- TP alone communicates too much for InfiniBand. Instead:
- Dense models: TP inside each node, PP between nodes (e.g.
TP8PP2). - MoE models: EP across nodes (e.g.
EP16), TP8PP2 gives lower per-user latency, EP16 gives higher throughput.
- Dense models: TP inside each node, PP between nodes (e.g.
Related but different
Context Parallelism (splitting the attention computation, not the weights) is rare for LLMs but essential for video generation, see Context Parallelism.