LLM Systems
Understanding KV Cache Compression
A systems view of why KV cache becomes the limiting resource in long-context inference and where compression actually helps.
Autoregressive generation looks compute-heavy, but serving a large language model is often a memory-management problem. Each decoded token adds keys and values for every transformer layer. The cache avoids recomputing the full prefix, yet grows with sequence length, batch size, layer count, and hidden dimension.
That growth changes the shape of the serving system. A request may fit comfortably during prefill and become expensive during a long decode. A large batch may maximize arithmetic utilization while exhausting accelerator memory. Compression is useful only when it improves this system-level trade-off.
Start with the memory equation
For a conventional multi-head attention model, cache size is approximately proportional to:
2 × layers × sequence length × KV heads × head dimension × bytes per value
The factor of two accounts for keys and values. Grouped-query and multi-query attention reduce the number of KV heads. Lower precision reduces bytes per value. Token eviction or merging reduces effective sequence length. These mechanisms operate on different terms of the same equation.
Four compression families
Lower precision
Quantizing cached keys and values is attractive because it preserves the attention pattern and changes storage directly. The engineering challenge is not only accuracy. Dequantization cost, kernel support, memory alignment, and mixed-precision boundaries can erase theoretical gains.
Fewer retained tokens
Eviction keeps a subset of positions according to recency, attention statistics, or learned importance. It can cut memory aggressively, but creates a semantic risk: information that looks unimportant at one step may become necessary later.
Shared or reduced representations
Low-rank methods, token merging, and latent-cache designs store a smaller representation and reconstruct what attention needs. They move work from memory capacity toward compute and introduce additional model-system coupling.
Architectural reduction
Grouped-query attention and related model choices reduce cache size at training time. These are often the cleanest serving optimizations, but they cannot be retrofitted into every deployed model without adaptation.
Measure the serving outcome
Compression should be evaluated with end-to-end metrics:
| Dimension | What to measure |
|---|---|
| Capacity | Concurrent sequences at target context |
| Latency | Time to first token and inter-token latency |
| Throughput | Tokens per second at realistic traffic |
| Quality | Task accuracy across context positions |
| Stability | Worst-case behavior, not only averages |
A 50% cache reduction is not automatically a 2× throughput gain. Attention kernels, scheduler policy, model weights, and host-device transfer may become the next bottleneck.
Design principle
Treat KV cache compression as a change to the serving architecture. Choose a method only after identifying whether the real constraint is capacity, bandwidth, latency, or cost. The best method is the one that moves the bottleneck without damaging the workload that matters.