There is a widespread misconception about why quantization speeds up artificial intelligence models. Many assume that performing 4-bit math simply takes fewer arithmetic clock cycles than 16-bit math., in modern large language model inference, the arithmetic unit is almost never the bottleneck.
The Arithmetic Intensity Trap
In standard autoregressive token generation, a GPU must load every single parameter of the model from High Bandwidth Memory (HBM) into on-chip SRAM cache just to calculate a single output token. The arithmetic intensity (operations per byte transferred) is approximately 1 FLOP per byte loaded.
Modern Tensor Core GPUs (like the NVIDIA H100) can perform nearly 2,000 trillion floating-point operations per second (2 PFLOPS), but their memory bus can only transfer 3.35 terabytes of data per second. The GPU cores spend over 90% of their time idling, waiting for memory bytes to crawl across the silicon bus.
Quantizing weights from FP16 (2 bytes) to 4 bits (0.5 bytes) shrinks the total data volume transferred across the memory bus by exactly 4x. The GPU spends 4x less time waiting for memory transfers, yielding an almost linear 3x to 4x speedup in tokens generated per second.
NVIDIA Blackwell and Native FP4 Tensor Cores
Previous architectures (Hopper, Ampere) could store INT4 weights on disk and in VRAM, but had to dequantize them back into FP16 inside the register file before executing matrix multiplication. This saved memory bandwidth, but did not accelerate compute execution.
The NVIDIA Blackwell architecture introduces native fifth-generation Tensor Cores with hardware-level FP4 micro-scaling support. Blackwell computes matrix multiply-accumulate directly on 4-bit floating point numbers inside the ALUs, delivering up to 20 PFLOPS of FP4 compute per chip, quadrupling inference throughput while cutting energy consumption per token in half.
Continue Your Exploration
This series translates the core principles of numerical precision, dynamic range, outlier feature clipping, and hardware Tensor Cores into accessible visual models. For full empirical telemetry, mathematical error bounds, and hardware benchmarking, consult our foundational monograph: Quantization Reasoning Degradation: 5 Empirical Rules.
Foundations of LLM Quantization & Low-Bit Inference
An empirical exploration through INT8/INT4/FP4 quantization, straight-through estimators, outlier features, and hardware tensor cores.