Chunk Overlap

Chunk overlap is the deliberate repetition of text at the boundary between consecutive chunks in a retrieval pipeline. When a document is cut into pieces for embedding, each cut lands somewhere, and wherever it lands it separates something that belonged together. Overlap means each chunk carries a tail of the previous one and a head of the next, so a sentence, a definition or a number that straddles a boundary survives intact in at least one of the two pieces. The failure it prevents is specific and easy to miss. A clause that begins in one chunk and completes in the next produces two fragments that each look reasonable and neither of which answers the question. Worse, the fragment that gets retrieved often reads as complete — a condition without its exception, a figure without the qualifier that follows it — so the model generates a confident answer from material that was accurate only in context. The retrieval metrics look fine. The answer is wrong. Overlap is not free. It inflates the index, it costs embedding compute proportional to the duplication, and it changes what the search returns: near-identical neighbouring chunks compete for the same slots, so the top results can be three versions of the same passage while the second-best source never surfaces. Deduplication after retrieval, or a reranking pass that penalises near-duplicates, is usually needed once overlap is anything but small. The amount is a trade-off rather than a constant, and it depends more on the shape of the text than on any general rule. Dense reference material, contracts, specifications and anything with cross-references benefit from more, because meaning is carried across sentences. Narrative prose and self-contained records need much less. The honest way to set it is to test both ends on a fixed evaluation set and look at whether answers improve, rather than reasoning about it in the abstract. Overlap is also a blunt fix for a problem better solved by cutting in the right places. Splitting on structure — headings, sections, list items, table rows — puts the boundaries where the document itself already has them, and a structure-aware splitter usually needs far less overlap than a fixed-size one to reach the same answer quality. Overlap then becomes insurance for the boundaries that structure could not predict. One detail worth attending to: whatever context a chunk needs in order to be understood alone should be attached rather than hoped for. A chunk that begins mid-argument with an unresolved pronoun is a retrieval liability regardless of overlap, and prefixing each chunk with its document title and section path is a cheaper improvement than widening the overlap window.