Document Chunking for RAG: What We Dropped and What We Ship

Fixed-Token Windows: Fast to Ship, Fast to Break

This is where almost every team starts, and for good reason. You pick 512 or 1000 tokens, add a 10-20% overlap, and you have an index by lunchtime. On a corpus of clean, uniform text — support articles, a knowledge base written by one team — it's genuinely fine. We still reach for it when the corpus warrants nothing more.

The problem shows up the moment documents have structure the tokenizer can't see. On one engagement with a corpus of supplier contracts, fixed windows cut a payment-terms table so that the header row ("Milestone / Amount / Due") landed in one chunk and the actual figures landed in the next. Retrieval pulled the numbers, the model had no idea what they referred to, and it confidently invented column labels. The demo query happened to hit a clean chunk. The user's real query didn't.

Overlap papers over this a little and papers over nothing structural. You're paying for duplicated tokens on every retrieval and still splitting a legal clause across a boundary the model can't reason across.

The demo query happened to hit a clean chunk. The user's real query didn't.

• →Use it when: documents are short, uniform, and prose-heavy with no tables or hierarchy.
• →Drop it when: your corpus has tables, numbered clauses, or code — the boundary will land in the worst possible place.

Naive Recursive Splitting: Better, Until Confluence

The obvious upgrade is recursive character splitting — try to break on paragraphs, then sentences, then words, respecting a max size. It's the default in most frameworks and it's a real improvement on blind token windows because it at least tries to keep semantic units intact.

It held up until we pointed it at a large Confluence export. Enterprise wikis are not clean prose. They're deeply nested bullet lists, collapsed macros, tables inside panels, half-finished pages, and the same policy documented three times with contradictory details. Recursive splitting saw a 3,000-word page with a two-level heading structure and no paragraph breaks worth speaking of, and produced chunks that started mid-list-item and ended mid-sentence. Worse, it had no idea that an H2 halfway down the page had changed the topic entirely — so a chunk about "expenses" carried no signal that it lived under the "Contractor Policy" heading rather than "Employee Policy."

The failure is subtle because retrieval still returns something plausible. You only catch it when you run real queries through an eval harness and watch the wrong policy get cited. If you haven't read why RAG demos break in production, chunk-boundary drift is the mechanism behind most of it.

Semantic Chunking: Elegant in the Notebook, Brittle in Production

So you get clever. Embed every sentence, measure the cosine distance between neighbours, and cut where the topic shifts. Chunks that respect meaning instead of arbitrary counts — it's the intellectually satisfying answer, and on a curated demo corpus it produces beautiful results.

In production it has two problems we couldn't design around. The first is speed and cost: you're running an embedding pass over every sentence just to decide boundaries, before you've embedded a single real chunk. On a corpus in the hundreds of thousands of documents that turns a re-index from an hour into most of a day, and re-indexing is not a one-time event. The second is determinism. The threshold that produced clean boundaries on one document type produced fifty-token slivers on dense legal text and 4,000-token monsters on rambling meeting notes. There is no single distance threshold that behaves across a heterogeneous corpus.

We kept a narrow version of it — semantic merging as a cleanup pass on top of structural splits — but semantic distance as the primary boundary decision didn't survive contact with a messy, changing corpus.

• →The hidden cost is re-index time, not query time — and enterprise corpora change weekly.
• →One distance threshold cannot serve contracts, wiki pages, and transcripts in the same index.

What We Ship: Structure-Aware Splitting With Rich Metadata

The approach that survives is boring and it works: split on the document's own structure first, then attach metadata that turns each chunk into something you can filter, rank, and cite. Documents already tell you where the boundaries are — heading hierarchy, clause numbering, table structure, page breaks. Parse that structure before you touch a tokenizer.

Concretely, that means a real parsing stage per document type. PDFs go through layout-aware extraction so tables stay intact and get serialised as a unit — often with the surrounding heading prepended so the figures never lose their referent. Contracts split on clause and sub-clause numbering, and each chunk carries its parent clause and defined-term references. Confluence pages split on heading hierarchy, and every chunk inherits the full heading path as metadata so "expenses" is stamped with "Contractor Policy > Reimbursement." Then we enforce a size band — merge tiny fragments up, split oversized sections down at the next structural seam, never in the middle of one.

The metadata is half the value. Source document, section path, effective date, document version, access-control tags — all of it travels with the chunk. That lets you filter to the current version of a policy instead of retrieving three contradictory copies, restrict retrieval by permission, and give the model something honest to cite. When you later need to cut token spend, structured chunks are also what make it possible to trim retrieval without regressing quality — you can drop or rerank on metadata instead of stuffing the context window.

Documents already tell you where the boundaries are. Parse that structure before you touch a tokenizer.

A Transferable Heuristic for Choosing Per Corpus

There is no universal chunk size, and anyone quoting you one hasn't looked at your documents. The decision is per corpus, and it comes down to two questions: how is this document structured, and how will people query it?

If the corpus has strong intrinsic structure — contracts, standards, technical docs with numbered sections — split on that structure and let chunk sizes vary within a band. If it's flat prose with weak structure — transcripts, emails, plain articles — recursive splitting with a moderate window is honestly fine, and reaching for anything cleverer is over-engineering. The query shape matters just as much: if people ask precise, lookup-style questions ("what's the notice period in the Acme contract"), you want smaller, tightly-scoped chunks and strong metadata filtering. If they ask broad, synthesis questions, you want larger chunks that preserve context, and you lean harder on the reranker.

Whatever you pick, the size is a hypothesis, not a decision. The only way to know it's right is to run real queries against it. Chunking changes belong in your eval harness alongside prompt and model changes — treat it as the deliverable, not an afterthought — because a chunking tweak that helps one query class routinely hurts another, and you will not see it by eyeballing outputs.

• →Strong structure (contracts, specs): split on hierarchy, variable size within a band, rich metadata.
• →Weak structure (transcripts, emails): recursive splitting, moderate window — don't over-engineer.
• →Precise lookup queries: smaller chunks, aggressive metadata filtering.
• →Broad synthesis queries: larger chunks, stronger reranking.

What's Still Hard

This isn't solved, and it's worth being honest about where it still hurts. Layout-aware PDF extraction falls over on scanned documents and anything with multi-column layouts or nested tables. We still hand-tune parsers per document family. Tedious work nobody enjoys. Detecting which of three near-duplicate policy pages is authoritative is a data-governance problem masquerading as a retrieval problem, and metadata only helps if someone maintains it. And re-chunking a live corpus without invalidating everything downstream — citations, cached retrievals, eval baselines — is genuinely fiddly.

What's changed for us is that we now treat chunking as a first-class engineering stage with its own tests, rather than a one-liner buried in an ingestion script. That reframing is most of the win. The teams that treat it as configuration ship RAG systems that break in month four; the teams that treat it as a parsing and metadata problem ship ones that hold up.

If you're staring at a corpus of messy PDFs and a wiki nobody trusts, start by classifying your documents by structure and your queries by shape before you write a single splitter. That's the audit that tells you which of these strategies you actually need.