Number Formats

Every low-precision format is a bet on how many bits go to exponent (dynamic range) versus mantissa (precision).

Anatomy

A float is $(-1{)}^s\cdot 2^{e-\text{bias}}\cdot 1.m$. With $E$ exponent bits and $M$ mantissa bits: $\text{dynamic range}\approx 2^{2^E},{\epsilon }_{\text{machine}}=2^{-(M+1)},\text{decimal digits}\approx 0.3(M+1)$

The table

Format	Bits (S/E/M)	Max finite	Smallest normal	Rel. precision $\epsilon$	Where used
FP32	1/8/23	3.4e38	1.2e−38	6.0e−8	accumulators, master weights, norms
TF32	1/8/10	3.4e38	1.2e−38	4.9e−4	tensor-core FP32 fallback
FP16	1/5/10	65504	6.1e−5	4.9e−4	legacy inference; needs loss scaling in training
BF16	1/8/7	3.4e38	1.2e−38	3.9e−3	default training + inference activation dtype
FP8 E4M3	1/4/3	448	1.6e−2	6.3e−2	weights & activations (precision-leaning)
FP8 E5M2	1/5/2	57344	6.1e−5	1.3e−1	gradients (range-leaning)
FP6 E3M2	1/3/2	28	0.25	1.3e−1	weight-only, block-scaled
FP4 E2M1	1/2/1	6	0.5	2.5e−1	block-scaled weights/activations
INT8	8 (int)	127	, (step $s$)	$s$	classic PTQ, per-channel scales
INT4	4 (int)	7	, (step $s$)	$s$	weight-only, group size 32–128

FP4 E2M1 representable magnitudes: $\{0,0.5,1,1.5,2,3,4,6\}$, 8 positive values. It is unusable without a per-block scale.

The range-vs-precision choice

• BF16 replaced FP16 in training because gradients span ~$10^{-10}$ to $10^2$: FP16’s 5 exponent bits underflow, forcing loss scaling; BF16 keeps FP32’s exponent and just throws away mantissa.
• E4M3 for forward, E5M2 for backward for the same reason at 8 bits: activations are bounded and want precision, gradients are heavy-tailed and want range.
• Below 8 bits neither is enough on its own, so the exponent moves outside the element into a shared block scale: see Microscaling Formats.

Rule of thumb

Exponent bits buy range, mantissa bits buy precision.