A kernel is a function that runs on the GPU. Fast inference is largely about picking or writing good kernels and cutting memory traffic between them.
Why fusion helps
Many inference ops are memory-bound: they read a tensor from HBM, do a little math, write it back. Running them as separate kernels means a full HBM round-trip each time. Kernel fusion merges them into one kernel so intermediate values stay in registers/SRAM, one read, one write. (See Arithmetic Intensity.)
Fusion
- Classic wins: fuse
norm + activation, or the whole attention pipeline (FlashAttention). - torch.compile does automatic fusion and can cache the compiled result (compilation takes minutes).
Kernel selection
- Different kernels win on different shapes and GPUs. GEMM (matmul) libraries: cuBLAS, CUTLASS, CuTe, DeepGEMM: test per model.
- Low precision unlocks faster Tensor Core kernels (2x FLOPS at FP8: see Quantization).