When a model is too large for one GPU, inference must span several devices. Tensor parallelism (see Model Parallelism) is the first choice inside a node, but it communicates on every layer and stops scaling once it crosses slower links. Pipeline serving splits the model across the layer depth instead, so GPUs only pass activations at stage boundaries.
The idea
Assign contiguous blocks of layers to different GPUs. A request flows GPU 0 to GPU 1 to GPU 2, each running its slice, with only point-to-point activation transfers between stages.
Tensor parallel vs pipeline for serving
- Tensor parallel: splits each layer, one collective per layer. Lowest latency, but needs fast intra-node interconnect (NVLink) and does not scale well across nodes.
- Pipeline: splits across layers, tiny communication. Scales across nodes and cheap networks, but a single request sees added stage-to-stage hops.
Keeping stages busy
- Like in training (see Pipeline Parallelism), an idle pipeline has a bubble.
- In serving, the stream of incoming requests naturally fills the pipeline: while stage 0 handles a new request, later stages work on earlier ones.
- Throughput stays high; the trade is a modest latency increase per request from traversing stages.