Tenstorrent NPU

Tenstorrent’s accelerators take a different structural bet from GPUs and TPUs: instead of one big matmul engine or many SIMT lanes, they use a 2D grid of small, self-contained Tensix cores tied together by an on-chip network, with data movement expressed explicitly in software. The same mesh abstraction extends over Ethernet to many chips, so a single-chip program and a multi-chip program look structurally similar.

Core idea

Every Tensix core has its own compute engines, its own local SRAM (used as a managed scratchpad, not a cache), and its own network routers. Tensors are streamed from core to core (and to DRAM) as packets over a Network-on-Chip (NoC), and the programmer decides where each tile lives and when it moves.

The Tensix core

A Tensix core is not a single processor but a small cluster of specialized blocks coordinated by five “baby” RISC-V cores:

• Two data-movement RISC-V cores drive the two NoC interfaces: one typically acts as a reader (pulling tiles into local SRAM) and one as a writer (pushing results out). They issue NoC transactions and manage the circular buffers.
• Three compute RISC-V cores run a three-stage pipeline over the math engines:
  • T0 (Unpack): drives the unpacker, reading tiles from SRAM into the source registers.
  • T1 (Math): issues instructions to the matrix and vector engines.
  • T2 (Pack): drives the packer, moving results back to SRAM.

The math blocks themselves are not processors and cannot branch; the RISC-V cores issue commands and synchronize them.

• Matrix engine (FPU): the workhorse for matmul, convolution, and elementwise/pooling ops. It reads operands from two source registers (SrcA, SrcB) and accumulates into a destination register set (Dst). Maximum internal accuracy is TF32-class (about 19 bits), with bf16 the common path.
• Vector engine (SFPU): a SIMD unit for activations, normalization, transcendentals, and other pointwise work. It operates through the Dst set and its own LReg vector registers (for example 32 elements wide on Wormhole/Blackhole), and supports up to fp32.
• Packer / unpacker: hardware format converters between SRAM storage and compute registers. Because they convert in hardware, tiles can sit in SRAM in a compact form (for example block floating-point) while the engines compute on bf16/fp32, saving bandwidth and power with no software decompression.

Registers are historical

SrcA/SrcB/Dst are named from the matrix engine, which predates the vector engine. Dst is the only register set exposed by the high-level compute API; the vector engine also reads and writes it. fp32_dest_acc_en selects 16-bit vs 32-bit Dst storage, trading register capacity for accumulation width.

Local SRAM and circular buffers

Each core has roughly 1 to 1.5 MB of local SRAM (“L1”). It is an explicitly managed scratchpad: nothing is fetched or evicted automatically, so the kernel author owns placement and lifetime.

Producer/consumer dataflow inside and between cores is coordinated through circular buffers (CBs): statically allocated FIFO queues in SRAM that pipeline tiles between the data-movement cores and the compute pipeline. The synchronization primitives are explicit:

• cb_reserve_back / cb_push_back: a producer reserves space, fills it, then publishes.
• cb_wait_front / cb_pop_front: a consumer waits for data, uses it, then frees the slot.

This lets the unpacker, math engine, and packer run concurrently on different tiles (double-buffering the Dst set), overlapping compute with NoC transfers so no stage idles waiting on another.

Tiles and the dataflow model

• The native unit of work is a 32x32 tile (1024 elements), internally split into 16x16 “faces”. Tiling to 32x32 keeps the engines and registers fully utilized.
• Data movement is first class: a kernel is written as explicit reads, compute, and writes, not as loads against a transparent cache.
• Compute and communication are scheduled together, which matches the dataflow shape of neural networks and keeps the Roofline Analysis question front-and-center: performance is about keeping every engine fed across the NoC.

Network-on-Chip (NoC)

• The on-chip fabric is a 2D torus mesh connecting all Tensix cores, with packet-based routing and wraparound links.
• Tensors are streamed core to core as NoC packets; the two NoC interfaces per core let reads and writes proceed in parallel.
• DRAM is accessed as banks hanging off the same NoC (GDDR6 on Wormhole and Blackhole), so a core reaches memory and its neighbors through one uniform mechanism. There is no monolithic memory controller funneling all traffic.

Ethernet scale-out

Selected cores on the die are Ethernet cores that carry the NoC protocol off-chip. Chip-to-chip links connect dies directly into a larger mesh with effectively zero software overhead: a remote core is addressed much like a local one, so the programming model extends off-chip into multi-chip and multi-host (“Galaxy”) systems.

• Wormhole added on-chip Ethernet (multiple 100 Gb lanes) specifically to make scale-out native.
• Blackhole widened this to 400 Gb-class links with roughly a terabyte per second of aggregate off-chip bandwidth.

This is the key contrast with switched GPU fabrics and with the TPU torus: scale-out is the same mesh, just larger.

Software stack

• TT-Metalium: the low-level C++ kernel and dataflow layer. You write data-movement kernels (for the NoC RISC-V cores: NoC transactions, CB management) and compute kernels (the unpack/math/pack pipeline over the FPU/SFPU), and you place kernels onto specific cores in the grid.
• TT-NN: a higher-level, PyTorch-like op library (tensors, common NN ops) built on Metalium, so most models do not require hand-written kernels.
• tt-forge / front-end compilers: an MLIR-based path that ingests models from PyTorch and other frameworks and lowers them onto the core grid and NoC, partitioning work across Tensix cores, assigning DRAM/L1 buffers, and generating the data-movement schedule.

Tenstorrent Neural Network Stack

Tensor Layout Types:

• row major
• Tile layout