A dataflow accelerator spatially maps the computation graph onto an array of cores with explicit scratchpads and DMA, replacing SIMT’s dynamic latency hiding with a statically scheduled pipeline.
SIMT vs dataflow
| SIMT GPU | Dataflow accelerator | |
|---|---|---|
| Parallelism | thousands of threads, dynamically scheduled | tens–thousands of cores, statically placed |
| Latency hiding | oversubscribe with warps (runtime) | software pipelining + double buffering (compile time) |
| On-chip memory | caches + shared memory | explicit scratchpad (SRAM), no cache |
| Data movement | implicit loads, hardware coalescing | explicit DMA descriptors |
| Between stages | round-trip through HBM (unless fused) | on-chip FIFO/NoC, producer → consumer directly |
| Control | per-thread PC, divergence | few control threads, no divergence concept |
| Who is responsible | hardware + programmer | compiler, almost entirely |
The bet: if the schedule is known ahead of time, you can delete the caches, the coherence, the schedulers, and the speculation, and spend the area and power on MACs and SRAM instead.
Spatial pipelining
Instead of running one kernel to completion over the whole tensor and writing to HBM, you split the graph across cores and stream tiles between them:
core0: load tile ─► core1: matmul ─► core2: bias+GELU ─► core3: matmul ─► core4: store
(DMA in) (no HBM round trip) (DMA out)
- Intermediates never touch HBM. This is Kernel Fusion taken to its architectural conclusion: fusion is the default, not an optimization.
- Steady-state throughput is set by the slowest stage, so the compiler must balance work across stages; an unbalanced pipeline idles everything else.
- Fill and drain cost tiles, which matters when the tensor is small.
Explicit memory: scratchpad + DMA
There is no cache, so nothing arrives by accident. Every byte on-chip got there because a DMA descriptor said so.
- Double buffering is mandatory, not an optimization: DMA tile into buffer B while compute consumes buffer A, with semaphores/barriers marking completion. Miss it and you serialize transfer and compute.
- Determinism is the payoff: no cache misses, so cycle counts are predictable and analytical Performance Modeling is unusually accurate.
- Multi-level scratchpads are common (per-core L1 SRAM, shared L2/on-package SRAM, then HBM/DRAM).
- Some designs have large enough total SRAM to hold model weights entirely on chip, which changes the roofline completely: arithmetic intensity is measured against SRAM bandwidth, not HBM.
Why compilers matter more
On a GPU, a mediocre schedule still runs, warps absorb the slack. On a dataflow chip, the schedule is the performance:
- Tiling, placement, routing, buffer sizing, and pipeline balancing are all compile-time decisions with no runtime recovery.
- The op set is fixed and narrow; anything unsupported falls off a cliff (host fallback or a slow generic path).
- Hence the heavy investment in MLIR-style stacks and kernel DSLs (Accelerator Compilers) and the practical reality that “supported model coverage” is the dominant product concern.
- Common failure mode: a model that is 90% supported runs at a fraction of peak because the 10% forces HBM round-trips between pipeline segments.
Where it wins and loses
Wins: static shapes, high fusion opportunity, quantized inference, energy per inference, deterministic latency (SLO-sensitive serving).
Loses: dynamic shapes and control flow, sparse/irregular access, rapidly changing model architectures, anything requiring a kernel the compiler doesn’t have.
Tip
The question to ask about any dataflow part is not “what is peak TOPS” but “what fraction of my graph does the compiler fuse into one on-chip pipeline?” Every HBM round-trip it fails to eliminate costs you the thing you bought the chip for.
Systolic Arrays · Accelerator Compilers · Kernel Fusion · Arithmetic Intensity · Latency Hiding