๐ŸŒฟ CuriousWinds

Foundations of LLM Quantization: INT8, INT4, & FP4

Physical mechanisms, mathematical boundary conditions, and architectural trade-offs in Foundations of LLM Quantization: INT8, INT4, & FP4.

Modern frontier large language models, whether spanning 8 billion, 70 billion, or 405 billion parameters, are massive collections of decimal numbers. In their native uncompressed format (typically 16-bit brain floating point, or BF16), each parameter consumes 2 bytes of high-bandwidth memory (VRAM). A 70-billion-parameter model thus demands over 140 gigabytes of raw video RAM merely to sit idle in memory, before allocating a single byte to user context or intermediate activation tensors.

01 NUMERICAL REPRESENTATION ยท BINARY FRACTIONING

The Anatomy of a Number

What is a Float?

In standard floating-point representation, a number is partitioned into three distinct binary fields: a sign bit, an exponent, and a mantissa (fraction). This allows computers to represent numbers across astronomical dynamic ranges, from subatomic fractions ($10^{-38}$) to cosmic magnitudes ($10^{38}$):

Binary Precision Comparison: FP16 vs INT4
Format Bit Width Structure (Sign / Exp / Mantissa) Discrete Values Memory per 70B Model
FP16 / BF16 16 bits 1 sign, 5/8 exp, 10/7 mantissa 65,536 ~140 GB
INT8 8 bits 1 sign, 7 integer magnitude 256 ~70 GB
INT4 / FP4 4 bits 1 sign, 3 magnitude (or 2 exp / 1 mantissa) 16 ~35 GB
02 DATA COMPRESSION ยท ALGORITHMIC REDUCTION

The Squeeze

From 65,536 Values to Exactly 16

Quantization is the algorithmic art of discarding 75% to 87.5% of binary data while preserving the semantic computations learned during billions of GPU training hours. When quantizing a layer to 4 bits, every single floating-point number must be forced into one of exactly 16 discrete bins (from -8 to +7 in signed INT4).

The standard affine quantization formula maps a continuous float $W$ to an integer $q$ via a scale factor $s$ and zero-point $z$:

q = clamp( round(W / s) + z, -8, 7 )
Where $s = (W_{max} – W_{min}) / 15$. Rounding introduces quantization error e = W – (q – z) * s.
03 NEURAL REDUNDANCY ยท NONLINEAR RESILIENCE

Why It Works

The Smooth Redundancy of Deep Nets

If you rounded the numbers in your bank account or a space shuttle flight trajectory to 16 buckets, catastrophe would follow instantly. Why do neural networks survive this brutal crushing? Because neural networks are overparameterized, non-linear ensembles. The vast majority of weights in a matrix are distributed normally around zero. Shifting an individual weight by +/- 0.005 barely alters the post-activation output vector when thousands of incoming connections are summed together.

However, as we will explore in the next chapters, this resilience is deceptive. While superficial grammar and vocabulary survive 4-bit compression effortlessly, deep recursive reasoning, mathematical chain-of-thought, and precise coding logic sit on precarious precision cliffs.

Key Takeaway: The 4-Bit Trade-Off

Quantization from FP16 to 4-bit reduces memory footprint and memory bus transit by exactly 4x, transforming a model that requires an entire $300,000 server rack into one that fits on a single workstation or smartphone. But compressing millions of continuous weights into 16 discrete levels creates subtle rounding ripples that compound catastrophically across deep transformer layers.

Continue Your Exploration

This series translates the core principles of numerical precision, dynamic range, outlier feature clipping, and hardware Tensor Cores into accessible visual models. For full empirical telemetry, mathematical error bounds, and hardware benchmarking, consult our foundational monograph: Quantization Reasoning Degradation: 5 Empirical Rules.

Curated Inquiry Series • 10 Investigations

Foundations of LLM Quantization & Low-Bit Inference

An empirical exploration through INT8/INT4/FP4 quantization, straight-through estimators, outlier features, and hardware tensor cores.

Primary Scholarly Provenance & Empirical Attribution
Card, S. K., Moran, T. P., & Newell, A. (1983). The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates; ACM SIGCHI Model of Human Processor & Saccadic Latency Foundations.
DOI: 10.1145/358896.358899