loading
loading
Attention that only looks at a subset of tokens instead of all of them, trading completeness for speed.
To dodge attention's quadratic cost, sparse patterns let each token attend to only some others. Sliding-window attention restricts each token to its last W neighbors (like 4k tokens), so cost grows linearly with length; other schemes mix local windows with a few global tokens. This is how models like Mistral handle long inputs cheaply, the tradeoff being that information has to hop through multiple layers to travel far across the context.
Plainly
Think of Sparse / Sliding-Window Attention as the brain part that guesses or decides. Attention that only looks at a subset of tokens instead of all of them, trading completeness for speed.
In practice
Use it when model choice, prompts, latency, cost, or quality affect the product result. In practice, define the owner, input, output, and failure mode before you rely on it.