🌿 CuriousWinds

Post-Training Quantization in Modern LLMs

Fast parameter calibration, outlier handling, and channel-wise scaling for zero-retraining model compression.

Training a 70-billion-parameter foundation model from scratch requires millions of dollars in compute, thousands of GPUs, and months of continuous runtime. When it comes time to deploy that model to edge devices or cost-effective inference clusters, retraining is rarely an option. Enter Post-Training Quantization (PTQ).

PTQ takes an already trained FP16 checkpoint and converts its weight matrices into 4-bit or 8-bit representations in a matter of minutes or hours, without needing access to the original massive pretraining corpus.

In This Article
2 Curated Inquiries • Computing Systems & Machine Cognition
01 1 · QUANTIZATION ENGINEERING · WEIGHT SCALING

The Mechanism

Calibration Datasets

Because weights cannot simply be scaled in a vacuum, PTQ requires a small calibration dataset (typically 128 to 512 representative text sequences). The calibration workflow proceeds in three distinct phases:

1
Forward Pass Profiling

Feed calibration sequences through the network in FP16, capturing the empirical distribution of activations at every layer.

2
Clipping Threshold Optimization

Calculate the optimal clipping range alpha for each matrix by minimizing Mean Squared Error (MSE) or Kullback-Leibler (KL) divergence between original and quantized activations.

3
Static Discretization

Lock the scale factors $s$ and zero points $z$, bake the integer tensors into disk storage, and deploy the lightweight weights.

02 2 · DATA DISTRIBUTION · CALIBRATION BIAS

The Calibration Trap

Distribution Shift

The fundamental vulnerability of PTQ lies in the calibration set. If you calibrate an LLM using standard Wikipedia text, the clipping thresholds optimize for conversational prose. If a user subsequently prompts the model with Python AST trees, medical genetics, or LaTeX algebraic proofs, the model encounters activations that fall completely outside its calibrated clipping boundaries. The result is instant saturation and erratic output.

While naive PTQ works reliably down to 8 bits (INT8), pushing PTQ down to 4 bits without second-order error compensation routinely shatters model reasoning, which is why advanced techniques like GPTQ, AWQ, and QAT were invented.

Continue Your Exploration

This series translates the core principles of numerical precision, dynamic range, outlier feature clipping, and hardware Tensor Cores into accessible visual models. For full empirical telemetry, mathematical error bounds, and hardware benchmarking, consult our foundational monograph: Quantization Reasoning Degradation: 5 Empirical Rules.

Curated Inquiry Series • 10 Investigations

Foundations of LLM Quantization & Low-Bit Inference

An empirical exploration through INT8/INT4/FP4 quantization, straight-through estimators, outlier features, and hardware tensor cores.

Primary Scholarly Provenance & Empirical Attribution
Card, S. K., Moran, T. P., & Newell, A. (1983). The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates; ACM SIGCHI Model of Human Processor & Saccadic Latency Foundations.
DOI: 10.1145/358896.358899