loading
loading
Stored attention keys/values from past tokens so the model doesn't recompute the whole context on every new token.
When generating text, each new token would normally re-run attention over the entire prompt — wasteful, since past tokens don't change. The KV cache saves the key and value vectors for tokens already processed, so each new token only computes its own and reuses the rest. This is what makes generation fast, but the cache grows with context length and eats GPU memory — it's often the real bottleneck for long-context serving, and the thing prompt caching persists between requests.
Plainly
Think of KV Cache as roads and power for the app city. Stored attention keys/values from past tokens so the model doesn't recompute the whole context on every new token.
In practice
Use it when local behavior needs to become a reachable, reliable deployed service. In practice, define the owner, input, output, and failure mode before you rely on it.