Static batching wastes the accelerator: the server waits to collect a batch, runs it to completion, and idles whenever requests finish at different times. Continuous batching (also called iteration-level or in-flight batching) instead makes scheduling decisions every single decode step.
The idea
The batch is reformed at each iteration. When a request finishes, its slot is freed immediately and a waiting request takes its place, all without draining the rest of the batch.
Why static batching wastes time
- Requests in a batch have different output lengths, so short ones finish early and their compute slots sit empty until the longest one is done.
- New arrivals wait for the current batch to fully complete before they can start.
How continuous batching fixes it
- After every token step, the scheduler removes finished sequences and admits queued ones.
- The GPU is kept near-full, so throughput rises sharply with little latency cost.
- It pairs with Paged Attention: variable-length sequences entering and leaving the batch need a KV cache that is not tied to fixed contiguous slots.
Prefill vs decode scheduling
New requests need a compute-heavy prefill, while ongoing requests need cheap decode steps. Mixing them naively lets a big prefill stall everyone’s decode. Schedulers either chunk long prefills or separate the phases (see Disaggregated Inference) to protect inter-token latency.
Payoff
Continuous batching is the single biggest throughput win in modern serving and is the core insight behind vLLM.
MAX_SLOTS = 4
waiting = [] # requests that arrived, not yet running
active = {} # slot -> request
free = list(range(MAX_SLOTS))
t = 0
# request: {"id", "arrive", "max_new", "gen", "done"}
def admit():
while free and waiting and waiting[0]["arrive"] <= t:
req = waiting.pop(0)
slot = free.pop()
req["gen"] = 0
active[slot] = req
def step():
global t
admit()
if not active:
return False
# ONE forward over the whole live batch
batch = list(active.items())
# fake_forward(batch) # real model would run here
finished = []
for slot, req in batch:
req["gen"] += 1 # sample + append one token
if req["gen"] >= req["max_new"]:
finished.append(slot)
for slot in finished:
active.pop(slot)["done"] = t
free.append(slot)
t += 1
return True
# --- drive it ---
for i in range(8):
waiting.append({"id": i, "arrive": i, "max_new": 5, "gen": 0, "done": None})
while step() or waiting:
if not active and waiting:
t = waiting[0]["arrive"] # jump to next arrival
continue
print(t, {s: active[s]["id"] for s in active})