๐ŸŒฟ CuriousWinds

NVIDIA Blackwell FP4 Tensor Cores & Memory Systems

Micro-scaling formats, sub-byte numeric representation, and hardware execution limits in modern GPUs.

There is a widespread misconception about why quantization speeds up artificial intelligence models. Many assume that performing 4-bit math simply takes fewer arithmetic clock cycles than 16-bit math., in modern large language model inference, the arithmetic unit is almost never the bottleneck.

In This Article
2 Curated Inquiries • Computing Systems & Machine Cognition
01 COMPUTATIONAL ARCHITECTURE ยท MEMORY LATENCY

The Arithmetic Intensity Trap

Memory-Bound Inference

In standard autoregressive token generation, a GPU must load every single parameter of the model from High Bandwidth Memory (HBM) into on-chip SRAM cache just to calculate a single output token. The arithmetic intensity (operations per byte transferred) is approximately 1 FLOP per byte loaded.

Modern Tensor Core GPUs (like the NVIDIA H100) can perform nearly 2,000 trillion floating-point operations per second (2 PFLOPS), but their memory bus can only transfer 3.35 terabytes of data per second. The GPU cores spend over 90% of their time idling, waiting for memory bytes to crawl across the silicon bus.

Why Quantization Actually Delivers Speed

Quantizing weights from FP16 (2 bytes) to 4 bits (0.5 bytes) shrinks the total data volume transferred across the memory bus by exactly 4x. The GPU spends 4x less time waiting for memory transfers, yielding an almost linear 3x to 4x speedup in tokens generated per second.

02 QUANTIZATION ENGINEERING ยท DEQUANTIZATION OVERHEAD

NVIDIA Blackwell and Native FP4 Tensor Cores

Previous architectures (Hopper, Ampere) could store INT4 weights on disk and in VRAM, but had to dequantize them back into FP16 inside the register file before executing matrix multiplication. This saved memory bandwidth, but did not accelerate compute execution.

The NVIDIA Blackwell architecture introduces native fifth-generation Tensor Cores with hardware-level FP4 micro-scaling support. Blackwell computes matrix multiply-accumulate directly on 4-bit floating point numbers inside the ALUs, delivering up to 20 PFLOPS of FP4 compute per chip, quadrupling inference throughput while cutting energy consumption per token in half.

Continue Your Exploration

This series translates the core principles of numerical precision, dynamic range, outlier feature clipping, and hardware Tensor Cores into accessible visual models. For full empirical telemetry, mathematical error bounds, and hardware benchmarking, consult our foundational monograph: Quantization Reasoning Degradation: 5 Empirical Rules.

Curated Inquiry Series • 10 Investigations

Foundations of LLM Quantization & Low-Bit Inference

An empirical exploration through INT8/INT4/FP4 quantization, straight-through estimators, outlier features, and hardware tensor cores.

Primary Scholarly Provenance & Empirical Attribution
Card, S. K., Moran, T. P., & Newell, A. (1983). The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates; ACM SIGCHI Model of Human Processor & Saccadic Latency Foundations.
DOI: 10.1145/358896.358899