September 30, 2026 · 8 min read

What Is Chunking in RAG

A retrieval system never reads your whole document. It reads the passages you cut it into, and chunking is that cut. Split too coarse and one chunk carries three ideas, so its embedding blurs and the right query misses. Split too fine and a passage loses the context that made it mean anything, the way a line reading the company revenue grew by 3% no longer says which company or which quarter. There is no universal size. LangChain base text splitter defaults to 4,000 characters, LlamaIndex sentence splitter to 1,024 tokens, because the right cut depends on your documents and your queries. Anthropic reported that adding a short generated context to each chunk before embedding cut the failure rate for the top 20 retrieved chunks by 35%, and by 67% once reranking was stacked on top. Retrieval quality does not start at the model. It starts at the cut.