loading
loading
The two phases of inference: reading the whole prompt at once (prefill), then generating one token at a time (decode).
Prefill processes your entire input prompt in a single parallel pass to build the KV cache — it's compute-bound and sets your TTFT. Decode then generates output token by token, each step memory-bandwidth-bound and reusing that cache — it sets your per-token streaming speed. They have totally different performance profiles, which is why a 50k-token prompt with a short answer is slow to start but fast to finish, and why optimizations like batching, caching, and speculative decoding each target one phase. Understanding this split explains almost every latency surprise in LLM serving.
Plainly
Think of Prefill vs Decode as roads and power for the app city. The two phases of inference: reading the whole prompt at once (prefill), then generating one token at a time (decode).
In practice
Use it when local behavior needs to become a reachable, reliable deployed service. In practice, define the owner, input, output, and failure mode before you rely on it.