๐ŸŒฟ CuriousWinds

GPTQ vs AWQ: Protecting Salient Weights in LLMs

Second-order error compensation, salient weight preservation, and low-bit matrix quantization.

When full Quantization-Aware Training is too expensive, but naive Post-Training Quantization causes logic to collapse, machine learning engineers turn to second-generation algorithmic techniques: GPTQ and AWQ. Both methods recognize that not all weights are created equal.

01 1: QUANTIZATION ENGINEERING ยท COLUMNAR REFINEMENT

GPTQ

The Inverse Hessian Error-Compensator

Introduced by Frantar et al. in 2022, GPTQ (Generalized Post-Training Quantization) builds upon the classical Optimal Brain Surgeon framework. Instead of quantizing all weights in parallel and hoping for the best, GPTQ quantizes weights column by column in a fixed order.

The Mathematical Mechanism of GPTQ

When a weight W_ij is rounded to its nearest 4-bit integer, it creates a small error delta = W_ij – q_ij. GPTQ calculates the Hessian matrix of the layer activations (H = 2 * X * X^T) and uses the inverse Hessian (H^-1) to immediately adjust all remaining unquantized weights in that row, actively canceling out the rounding distortion before moving to the next column.

02 2: ACTIVATION-AWARE OPTIMIZATION ยท MAGNITUDE-PRESERVING

AWQ

Activation-Aware Weight Sparing

While GPTQ focuses on weight-level error propagation, AWQ (Activation-aware Weight Quantization), pioneered by Lin et al. in 2023, introduced a profound insight: Weights that connect to high-magnitude activations are exponentially more important than weights that connect to near-zero activations.

How AWQ Protects the 1% Salient Weights

AWQ inspects activation magnitudes across a small calibration sample to identify the top 1% most salient weight channels. Instead of keeping them in memory-heavy FP16, AWQ calculates an per-channel scaling factor $s > 1$ that mathematically inflates the salient weights prior to quantization, reducing their relative quantization error to negligible levels.

03 3: QUANTIZATION COMPARATIVE ยท SALIENCY METRICS

Direct Comparison

GPTQ vs AWQ
Feature GPTQ AWQ
Optimization Target Minimizes inverse Hessian reconstruction error Protects salient weights based on activation norms
Quantization Speed ~2-4 hours for 70B model ~15-30 minutes for 70B model
Hardware Inference Speed High (custom CUDA/Triton kernels) Highest (zero overhead channel scaling)

Continue Your Exploration

This series translates the core principles of numerical precision, dynamic range, outlier feature clipping, and hardware Tensor Cores into accessible visual models. For full empirical telemetry, mathematical error bounds, and hardware benchmarking, consult our foundational monograph: Quantization Reasoning Degradation: 5 Empirical Rules.

Curated Inquiry Series • 10 Investigations

Foundations of LLM Quantization & Low-Bit Inference

An empirical exploration through INT8/INT4/FP4 quantization, straight-through estimators, outlier features, and hardware tensor cores.

Primary Scholarly Provenance & Empirical Attribution
Card, S. K., Moran, T. P., & Newell, A. (1983). The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates; ACM SIGCHI Model of Human Processor & Saccadic Latency Foundations.
DOI: 10.1145/358896.358899