Models are trained with weights and activations in a native format, usually BF16 or FP16. Quantization converts those values to a lower-precision format, cutting both the compute and the memory each value costs. It is the one technique that helps both phases of inference at once, but it is also the only one that can silently degrade output quality.
Why it speeds up both phases
- Prefill (compute-bound): lower precision runs on Tensor Cores with roughly twice the FLOPS.
- Decode (memory-bound): each value loads half as many bytes, effectively doubling memory bandwidth.
Overhead means it is not a clean 2x. In practice, dropping one level of precision buys 30–50% better performance for LLMs (see Inference Metrics).
Number formats
- Inference lives in the 16, 8, and 4-bit range. FP64/FP32 are for training or scientific work; FP6 is still experimental.
- A format is defined by its precision (bit count), type (integer vs floating point), and scale factor (multiplier back to high precision).
- These set the two things that matter: dynamic range (spread from smallest to largest representable value) and granularity (how many values share one scale factor).
Why floating point beats integer
Floating-point formats (sign + exponent + mantissa, e.g. FP8 as E4M3) have far higher dynamic range than integer formats, so they represent the outlier values that matter most in inference. For production, stick to float formats, INT8/INT4 are fine for local/edge (GGUF, Unsloth’s 1.58-bit) but not for quality-sensitive serving.
- Granularity levels: per-tensor (one scale factor), per-channel, or per-block. Finer granularity preserves outliers but costs memory and compute to store and apply scale factors.
- Microscaling formats (MXFP8, MXFP4 on Blackwell) compute a blockwise scale on every 32 values; NVFP4 goes finer still (block size 16 plus a global scale). Blackwell applies scale factors in Tensor Cores to hide the overhead.
What to quantize
Components differ in sensitivity, from least to most risky:
- Weights (linear layers), least sensitive, biggest and safest win.
- Activations: somewhat sensitive.
- KV cache: moderately sensitive, but quantizing it boosts Prefix Caching and Disaggregated Inference by fitting and moving more KV Cache.
- Attention: highly sensitive; softmax and similar are almost always left in native precision because errors compound token to token.
The sensible default
FP8 (ideally a microscaling format like MXFP8) on select linear layers, activations, and often the KV cache, leaving attention and the input/output layers untouched. FP8 is the sweet spot for performance with no perceptible quality loss.
Approaches and quality checks
- Quantization-aware training (QAT): scale factors learned during training (e.g. GPT-OSS in MXFP4). Only labs can do this.
- Post-training quantization (PTQ): convert finished weights with calibration: what inference engineers actually do with open models. NVIDIA TensorRT Model Optimizer (ModelOpt) is a leading tool, and its output runs on vLLM, SGLang, and TensorRT-LLM.
- Verify with perplexity, intelligence benchmarks (MMLU, SWE-bench), and custom evals. The bar is a difference indistinguishable from run-to-run noise.
Quantization is a scale, not a switch
If a domain can’t risk quality, quantize less: FP8 instead of FP4, or weights-only. Every other technique in this folder is lossless, quantization is the only one that trades quality for speed.