Sliding Window Attention

Sliding window attention restricts each token to attending over a fixed-size neighbourhood of recent tokens rather than the whole sequence. Standard attention compares every token with every other, so cost and memory grow with the square of sequence length — the reason long contexts are expensive to serve. A window of, say, a few thousand tokens turns that into linear growth, because the work per token stops depending on how long the document is. Information still travels further than one window: with each layer attending inside its own window, a token can be influenced by material several windows away, in the way a stack of small convolutions covers a wide receptive field. Many long-context models therefore interleave windowed layers with a few full-attention layers, so most of the sequence is processed cheaply while some layers retain a global view. The trade-off is worth stating plainly for anyone building on top. Windowing reduces the cost of long inputs but weakens exact recall of a detail sitting far outside the current window, which is precisely the pattern that makes a model look unreliable on "find the one clause in this contract" tasks. If your workload depends on pinpoint retrieval from a large document, a retrieval step that puts the relevant passage near the question is more dependable than trusting any attention scheme to reach across the whole input.