Transformer FLOPs and Params

Parameters per block

With model dim $d$, $n_h$ query heads, $n_{kv}$ KV heads, head dim $d_{\text{head}}$ (usually $n_h{d}_{\text{head}}=d$), FFN hidden $d_{ff}$:

$P_{\text{attn}}=\underset{W_Q}{\underbrace{d^2}}+\underset{W_K,W_V}{\underbrace{2d{n}_{kv}{d}_{\text{head}}}}+\underset{W_O}{\underbrace{d^2}}$ $P_{\text{ffn}}=\{\begin{array}{ll}2d{d}_{ff}& \text{vanilla}\\ 3d{d}_{ff}& \text{gated (SwiGLU)}\end{array}$

Classic case (for MHA, $d_{ff}=4d$): $P_{\text{attn}}=4d^2$, $P_{\text{ffn}}=8d^2$, so $N\approx 12L{d}^2(+Vd\text{ per embedding table})$ LayerNorm/RMSNorm and biases are $O(d)$. Embeddings are not negligible for small models: an 8B model with $V=128$K and untied embeddings carries $2\cdot 128\text{K}\cdot 4096\approx 1.05$B params, 13% of the total.

FLOPs per token

Every parameter participates in one multiply-accumulate = 2 FLOPs: $\text{forward}\approx 2N\text{ FLOPs/token}\text{fwd+bwd}\approx 6N\text{ FLOPs/token}$ Backward is 2× forward (grad w.r.t. input + grad w.r.t. weights). Activation checkpointing pushes this to $\approx 8N$.

Attention vs FFN

The $2N$ term covers the four projections and the FFN, all $O(T)$. The score/value matmuls are $O(T^2)$ and carry no parameters: ${\text{FLOPs}}_{\text{attn-mm}}=2Ld{T}^2\text{(causal; }4Ld{T}^2\text{ unmasked)}$ Set it equal to the weight FLOPs $2NT=24L{d}^{2}T$: $\frac{2Ld{T}^2}{24L{d}^{2}T}=\frac{T}{12d}\Rightarrow \boxed{T>12d}$ So for $d=4096$ attention only dominates FLOPs past ~49K tokens; for $d=8192$, ~98K. Below that, prefill is a GEMM problem, not an attention problem.

Caveat: this is about FLOPs in prefill. At decode attention dominates time far earlier, because reading the KV cache is memory-bound while the weights are amortized over the batch.

MFU and MBU

$\text{MFU}=\frac{{\text{FLOPs}}_{\text{model}}/\text{s}}{{\text{FLOPS}}_{\text{peak}}}=\frac{2N\cdot \text{tok/s}}{{\text{FLOPS}}_{\text{peak}}}(\text{inference})$ Use $6N$ for training. Prefill MFU of 0.35–0.5 is healthy; decode MFU of 1–5% is normal and not a bug: decode is bandwidth-limited, so measure MBU instead: $\text{MBU}=\frac{(W+BS{k}_{\text{tok}})/\text{TPOT}}{{\text{BW}}_{\text{peak}}}$ Good decode systems hit MBU 0.6–0.8. If both MFU and MBU are low, you are latency-bound: kernel launches, collectives, or Python overhead.

Rule of thumb

$N\approx 12L{d}^2$, forward $=2N$/token, training $=6N$/token, attention matters past $T\approx 12d$. Report MFU for prefill and MBU for decode, quoting MFU on a decode benchmark signals you don’t know which roof you’re under.