๐ŸŒฟ CuriousWinds

FP4 vs INT4 Quantization in Neural Reasoning

Physical mechanisms, mathematical boundary conditions, and architectural trade-offs in FP4 vs INT4 Quantization in Neural Reasoning.

Once you accept that neural network parameters must be compressed into 4-bit representations (providing exactly 16 discrete numerical states), a central architectural question emerges: How should those 16 states be distributed across the number line?

This decision splits modern machine learning engineering into two rival camps: INT4 (Uniform Integer Quantization) and FP4 (Floating-Point Quantization).

In This Article
3 Curated Inquiries • Computing Systems & Machine Cognition
01 1: QUANTIZATION ENGINEERING ยท DISCRETE GRADIENTS

INT4

The Rigidity of Uniform Grids

Integer quantization lays down 16 tick marks separated by strictly equal distances. If your scale factor $s = 0.5$, your 16 states are:

-4.0, -3.5, -3.0, -2.5, -2.0, -1.5, -1.0, -0.5, 0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5

The virtue of INT4 is computational simplicity: integer addition and multiplication logic in silicon uses roughly 5x less energy and silicon area than floating-point logic. But the drawback is structural mismatch: neural network weights do not inhabit a uniform grid. The vast majority of weights cluster tightly around zero, while rare weights extend far into the tails. INT4 wastes precious bins on regions of the number line where virtually no weights exist, while offering inadequate resolution at the center.

02 2: ARITHMETIC PRECISION ยท LINEAR QUANTIZATION

FP4

Non-Linear Logarithmic Density

Floating-point 4-bit (specifically the FP4 E2M1 format standardized by NVIDIA Blackwell and OCP) allocates its 4 bits differently: 1 sign bit, 2 exponent bits, and 1 mantissa bit. Because of the exponential field, the 16 values are distributed logarithmically:

FP4 E2M1 Discrete Level Distribution
Negative: -6.0, -4.0, -3.0, -2.0, -1.5, -1.0, -0.5, -0.0
Positive: +0.0, +0.5, +1.0, +1.5, +2.0, +3.0, +4.0, +6.0

Notice the dense clustering near zero (+/- 0.5, +/- 1.0, +/- 1.5) combined with expansive reach into the extremes (+/- 4.0, +/- 6.0). This logarithmic density profile closely mirrors the bell curve of trained transformer weights.

03 3: FLOATING-POINT ARCHITECTURE ยท EXPONENTIAL SPACING

The Micro-Scaling Revolution (MX-FP4)

To overcome the limitations of pure FP4, modern hardware utilizes micro-scaling (MX). Rather than sharing one scale factor across an entire million-weight matrix, weights are partitioned into tiny blocks of 16 or 32 elements. Each microscopic block receives its own 8-bit scale factor (E8M0). This micro-scaling eliminates the outlier penalty: if a single outlier appears in block 42, only block 42 expands its dynamic range, while neighboring blocks retain razor-sharp precision.

Continue Your Exploration

This series translates the core principles of numerical precision, dynamic range, outlier feature clipping, and hardware Tensor Cores into accessible visual models. For full empirical telemetry, mathematical error bounds, and hardware benchmarking, consult our foundational monograph: Quantization Reasoning Degradation: 5 Empirical Rules.

Curated Inquiry Series • 10 Investigations

Foundations of LLM Quantization & Low-Bit Inference

An empirical exploration through INT8/INT4/FP4 quantization, straight-through estimators, outlier features, and hardware tensor cores.

Primary Scholarly Provenance & Empirical Attribution
Card, S. K., Moran, T. P., & Newell, A. (1983). The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates; ACM SIGCHI Model of Human Processor & Saccadic Latency Foundations.
DOI: 10.1145/358896.358899