loading
loading
A small fast model guesses several tokens ahead, the big model verifies them in one pass.
Decoding is normally one token at a time, which is slow because each step needs a full forward pass. Speculative decoding has a cheap 'draft' model propose a chunk of tokens, then the real model checks them all in a single pass — accepted guesses are free speed, rejected ones fall back to normal. Output is identical to the big model alone; you just get it 2-3x faster. It's a serving-side optimization you usually don't see as a user, but it's why some providers stream so fast.
Plainly
Think of Speculative Decoding as roads and power for the app city. A small fast model guesses several tokens ahead, the big model verifies them in one pass.
In practice
Use it when local behavior needs to become a reachable, reliable deployed service. In practice, define the owner, input, output, and failure mode before you rely on it.