What is the KV cache and why is it critical for LLM inference?
AI infra interviews in 2026 expect this. Without KV cache understanding, you can't discuss serving latency or memory sizing.
Transformer attention at each step reads all prior tokens' keys and values. Naively regenerating them every step is O(n²) per step.
KV cache stores each layer's K and V tensors as tokens are processed. Generation of token t+1 only needs to compute K, V for token t, then attend over the cache.
Memory: cache size = 2 × num_layers × num_heads × head_dim × seq_len × batch × precision. A 70B model with 4K context can eat tens of GB per request.
Serving optimizations: PagedAttention (vLLM) treats KV memory like OS virtual memory pages — avoids fragmentation. FlashAttention-2 fuses compute.
Trade-offs: longer context = more cache. Quantization of cache (fp16 → int8/int4) cuts memory 2-4x at some quality cost.
In autoregressive generation each new token's attention needs the keys and values of all prior tokens. Without caching, every step recomputes every prior K/V — quadratic. KV cache says: once computed, keep them; at step t+1 you only compute K, V for the newly generated token and attend over the stored cache. This turns generation from O(n²) per step to O(n) per step. The downside is memory — 2 tensors × all layers × all heads × head_dim × seq_len × batch. For a 70B model with 4K context you can blow past 10GB per request. That's why vLLM introduced PagedAttention, which pages the KV cache like OS virtual memory so you don't waste slots, enabling high concurrency on one GPU. KV cache quantization (int8 or even int4) is the next lever when you're memory-bound.
KV cache is why first-token latency (prefill) and per-token latency (decode) scale differently — prefill is compute-bound, decode is memory-bound.
Sliding-window attention (Mistral) keeps KV cache size bounded even with long contexts.
HireMe AI (即答侠) ties KV cache to concrete numbers (MB per token for Llama 70B) so answers feel engineering-grade.
Recomputing is quadratic per step — for 4K context that's 16M extra ops per token vs reading a cache. Latency would be unusable.
Yes — prefix caching shares common system-prompt KV across users. Big win for chat apps with shared system messages.
HireMe AI listens to your interview in real-time, detects questions, and generates instant answer suggestions — so you never freeze up again.
Try HireMe AI Free