🌿 CuriousWinds

KV Cache Quantization: FP8, INT4, & Context Scaling

Memory scaling, sub-4-bit numerical precision, and attention accuracy trade-offs in long-context inference.

When running inferences on large language models, the model’s static weights are only half of the memory equation. In modern production environments where prompts contain hundreds of pages of documentation, legal contracts, or multi-turn dialogues, a hidden memory hog quickly dwarfs the model weights: the Key-Value (KV) Cache.

In This Article
2 Curated Inquiries • Computing Systems & Machine Cognition
01 1 · MEMORY ARCHITECTURE · CACHE COMPRESSION

The KV Cache Problem

Quadratic Memory Growth

Autoregressive transformers generate output token by token. To prevent recomputing the attention vectors of every prior token on every generation step, GPUs cache the Key and Value matrices in high-speed VRAM. The memory required scales linearly with sequence length $L$, batch size $B$, number of layers $N_{layers}$, and hidden dimension $d$:

KV_Memory = 2 imes B imes L imes N_{layers} imes N_{heads} imes d_{head} imes ext{bytes_per_element}

For a 70B model (like Llama-3-70B) serving a single user at 128,000 tokens in standard 16-bit FP16, the KV cache alone demands over 80 gigabytes of VRAM, more than the model weights themselves! Serving 10 concurrent users becomes financially impossible.

02 2 · TRANSFORMER CIRCUITS · QUANTIZATION ENGINES

Quantizing the Cache

FP8 and INT4

To overcome this context ceiling, modern inference engines (such as vLLM, TensorRT-LLM, and SGLang) implement dynamic KV-Cache Quantization:

FP8 KV Cache (Lossless Industry Standard)

Compresses the cache from 16 bits to 8 bits (E4M3 or E5M2). Halves VRAM usage with virtually zero measurable degradation across needles-in-a-haystack retrieval or multi-document summarization.

INT4 KV Cache (Aggressive High-Density)

Compresses the cache by 4x. Requires per-head or per-token dynamic scaling factors to preserve attention softmax peaks, unlocking 128k context on consumer GPUs.

By pairing 4-bit model weights with 4-bit or 8-bit KV caching, engineers reduce the total memory profile of a frontier LLM by nearly 80%, allowing long-context generation to run on single-chip hardware.

Continue Your Exploration

This series translates the core principles of numerical precision, dynamic range, outlier feature clipping, and hardware Tensor Cores into accessible visual models. For full empirical telemetry, mathematical error bounds, and hardware benchmarking, consult our foundational monograph: Quantization Reasoning Degradation: 5 Empirical Rules.

Curated Inquiry Series • 10 Investigations

Foundations of LLM Quantization & Low-Bit Inference

An empirical exploration through INT8/INT4/FP4 quantization, straight-through estimators, outlier features, and hardware tensor cores.

Primary Scholarly Provenance & Empirical Attribution
Card, S. K., Moran, T. P., & Newell, A. (1983). The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates; ACM SIGCHI Model of Human Processor & Saccadic Latency Foundations.
DOI: 10.1145/358896.358899