Chunking

Chunking is the process of splitting a large document or body of text into smaller, more manageable pieces ("chunks") before it's processed by an AI pipeline — most commonly as a preparatory step for retrieval-augmented generation (RAG), where documents must be broken into chunks small enough to be individually embedded, indexed in a vector database, and retrieved by relevance, but large enough that each chunk retains enough surrounding context to be independently meaningful when later inserted into a prompt without its original document context. Chunking strategy has an outsized, often underestimated impact on RAG system quality: chunk too small (a single sentence or paragraph fragment), and retrieved chunks frequently lack the surrounding context needed to answer a question fully or unambiguously, even when the retrieval step correctly identifies the relevant general area of a document; chunk too large (an entire multi-page document section), and the embedding representing that chunk becomes a diluted average of many different sub-topics, making semantic retrieval less precise, while also wasting context-window budget on irrelevant surrounding material once retrieved into a prompt (directly connecting to the context-stuffing problem). Common chunking strategies include fixed-size chunking (splitting every N tokens, simple but naively cuts across sentence or paragraph boundaries mid-thought), recursive/structural chunking (splitting along natural document boundaries — paragraphs, then sentences if a paragraph is still too large — preserving semantic coherence better than fixed-size splitting), semantic chunking (using embedding similarity between adjacent sentences to detect genuine topic boundaries and split there rather than at an arbitrary size), and chunking with overlap (each chunk includes a small overlapping window of text from the adjacent chunk, so information near a chunk boundary isn't lost or orphaned from its context on either side). For SaaS builders building any AI feature over a document corpus — a documentation Q&A bot, a contract-analysis tool, a knowledge-base search assistant — chunking strategy is one of the highest-leverage, most commonly under-invested-in levers for RAG quality, frequently mattering more to end-to-end accuracy than which embedding model or which LLM is used downstream. Concrete worked example: a documentation-chatbot SaaS tool initially chunks its help-center articles by fixed 500-token windows with no regard for structure, and users report the bot frequently gives incomplete answers to procedural "how do I..." questions — because a fixed-size chunk boundary often falls in the middle of a numbered step list, retrieving only steps 1-3 of a 6-step process while step 4-6 land in the next chunk, which isn't retrieved. Switching to structural chunking (splitting along markdown headers and keeping each complete numbered list or procedure intact within one chunk, with 50-token overlap at boundaries) measurably improves answer completeness on their evaluation set, without any change to the embedding model, retriever, or generation prompt — a clear demonstration that chunking strategy alone was the accuracy bottleneck.