loading
loading
Reusing the model's precomputed state for a repeated prompt prefix so you don't pay to process it again.
When many requests share the same long prefix — a big system prompt, a document, a codebase — the model can cache the internal state (the KV cache) from processing it once and reuse it, skipping the expensive prefill on later calls. That slashes both TTFT and cost on the cached portion, often 10x cheaper for those tokens. Put your stable, bulky content (instructions, docs) at the front and the variable part last to maximize hits. Provider-side this is the serving mechanism; the billed feature is usually called prompt caching.
Plainly
Think of Context Caching as roads and power for the app city. Reusing the model's precomputed state for a repeated prompt prefix so you don't pay to process it again.
In practice
Use it when local behavior needs to become a reachable, reliable deployed service. In practice, define the owner, input, output, and failure mode before you rely on it.