When full Quantization-Aware Training is too expensive, but naive Post-Training Quantization causes logic to collapse, machine learning engineers turn to second-generation algorithmic techniques: GPTQ and AWQ. Both methods recognize that not all weights are created equal.
GPTQ
Introduced by Frantar et al. in 2022, GPTQ (Generalized Post-Training Quantization) builds upon the classical Optimal Brain Surgeon framework. Instead of quantizing all weights in parallel and hoping for the best, GPTQ quantizes weights column by column in a fixed order.
When a weight W_ij is rounded to its nearest 4-bit integer, it creates a small error delta = W_ij – q_ij. GPTQ calculates the Hessian matrix of the layer activations (H = 2 * X * X^T) and uses the inverse Hessian (H^-1) to immediately adjust all remaining unquantized weights in that row, actively canceling out the rounding distortion before moving to the next column.
AWQ
While GPTQ focuses on weight-level error propagation, AWQ (Activation-aware Weight Quantization), pioneered by Lin et al. in 2023, introduced a profound insight: Weights that connect to high-magnitude activations are exponentially more important than weights that connect to near-zero activations.
AWQ inspects activation magnitudes across a small calibration sample to identify the top 1% most salient weight channels. Instead of keeping them in memory-heavy FP16, AWQ calculates an per-channel scaling factor $s > 1$ that mathematically inflates the salient weights prior to quantization, reducing their relative quantization error to negligible levels.
Direct Comparison
| Feature | GPTQ | AWQ |
|---|---|---|
| Optimization Target | Minimizes inverse Hessian reconstruction error | Protects salient weights based on activation norms |
| Quantization Speed | ~2-4 hours for 70B model | ~15-30 minutes for 70B model |
| Hardware Inference Speed | High (custom CUDA/Triton kernels) | Highest (zero overhead channel scaling) |
Continue Your Exploration
This series translates the core principles of numerical precision, dynamic range, outlier feature clipping, and hardware Tensor Cores into accessible visual models. For full empirical telemetry, mathematical error bounds, and hardware benchmarking, consult our foundational monograph: Quantization Reasoning Degradation: 5 Empirical Rules.
Foundations of LLM Quantization & Low-Bit Inference
An empirical exploration through INT8/INT4/FP4 quantization, straight-through estimators, outlier features, and hardware tensor cores.