No single parallelism strategy scales a large model on its own. Real training runs combine several axes at once. The classic combination is data plus tensor plus pipeline (3D); adding context and expert parallelism gives the full 5D picture. The goal is to pick degrees on each axis so every device stays busy and no interconnect is overloaded.
The five axes
- Data Parallelism and ZeRO and FSDP: replicate or shard across the batch.
- Model Parallelism: tensor and sequence parallel, split within a layer.
- Pipeline Parallelism: split across layer depth into stages.
- Context Parallelism: split along the sequence for long context.
- Expert Parallelism: split experts for MoE. Total devices = the product of all the degrees.
How the axes map to hardware
Each axis has a different communication profile, so each belongs on a different tier of the network.
| Axis | Communicates | Volume | Place it |
|---|---|---|---|
| Tensor / sequence | Every layer, on critical path | High | Inside a node (NVLink) |
| Expert | All-to-all per MoE layer | High | Fast-interconnect group |
| Context | KV blocks around a ring | Medium | Fast group |
| Pipeline | Activations at stage boundaries | Low | Across nodes |
| Data / ZeRO | Gradients once per step | Low, overlappable | Outermost, across nodes |
Rule of thumb
Put the chattiest axes on the fastest links. Fill a node with tensor (and expert) parallelism, span nodes with pipeline stages, and wrap data parallelism (with ZeRO sharding) around the outside where its gradient all-reduce can overlap with compute.
Choosing a configuration
- Set tensor-parallel degree to fit a layer, capped by the node size.
- Add pipeline stages until the model fits across nodes, keeping micro-batches high enough to shrink the bubble.
- Turn on ZeRO / FSDP and activation recomputation to fit what remains.
- Use leftover devices for data parallelism to raise throughput.
- Add context or expert parallelism only if long context or MoE demands it.
It is a search, not a formula
The best split depends on model shape, batch size, sequence length, and the exact interconnect topology. Expect to profile a few candidates rather than trust a closed-form answer.