Chunking Is a Data Modeling Decision, Not a Preprocessing Step
Pull-quote: “No one designs a database by slicing every table into 512-row pages and hoping the joins survive. Fixed-size chunking does exactly that to documents, and then we act surprised when retrieval returns half an answer.”
Why this matters
Chunking gets treated as a preprocessing detail: pick a splitter, pick a size, move on. It is closer to schema design. Chunk boundaries decide what your retriever can ever return as a unit, what context ships alongside a match, and what gets permanently separated from the thing it depends on. Embedding models and rerankers can only rank the units they are given. If the unit is wrong, everything downstream is tuning around damage that already happened.
The test is simple: for each document class you ingest, can you name the unit a correct answer lives in? If you cannot, the splitter is choosing your data model for you, and it is choosing by character count.
What fixed-size splitting destroys
In contracts, an obligation and the exception that modifies it are routinely separated, so the retriever returns the rule without the carve-out. In regulatory filings, sections bleed into each other mid-sentence and a risk disclosure inherits words from the section before it. In technical manuals, a warning drifts away from the procedure step it governs, which is the single worst place to lose context. Tables fare worst of all: a value cut from its header row is a number with no meaning.
| Document class | Natural retrieval unit | What naive splitting breaks | Structure-aware rule |
|---|---|---|---|
| Contracts | Clause, with its parent path | Exceptions land two chunks away from their obligations | Split on clause boundaries; carry section path and defined terms |
| Regulatory filings | Item or section | Sections bleed into each other mid-sentence | Split on item headings; tag filer, period, and section |
| Technical manuals | Procedure step with its warnings | Cautions detach from the steps they govern | Keep steps atomic; glue warnings to their steps |
| Research papers | Section, table, or figure | Methods mix into results; captions orphan | Split by section; keep captions with their objects |
Structure first, size second
The operating rule: parse before you split. Recover the document tree first, meaning headings, numbering, clause hierarchy, tables, and captions. Split on structural boundaries. Only then subdivide units that exceed your size budget, and do it within the unit, never across it.
Quality manual (rev 12)
│ parse structure first
├─ 8. Operation
│ ├─ 8.5 Production control ──► chunk {path: 8.5, rev: 12, effective_date}
│ │ ├─ 8.5.1 Controlled conditions ──► chunk {path: 8.5.1, rev: 12, effective_date}
│ │ └─ Table: process parameters ──► atomic chunk, headers attached
│ └─ 8.6 Release of products ──► chunk {path: 8.6, rev: 12, effective_date}
└─ 9. Performance evaluation ...Two patterns carry most of the weight. Parent-child chunks: match on the small unit, then hand the model the parent section when it needs surrounding context, so precision and context stop competing. Metadata that rides along: every chunk carries document id, section path, revision, and effective date. That metadata looks like bookkeeping now; it becomes your filter vocabulary at query time, and filters are the cheapest relevance win in the stack.
Where this shows up in the field
Quality management manuals are the clearest case. Engineers and auditors cite clauses, so chunks should align to clauses, and a table should never travel without its headers. Biomedical literature makes the same point for graph-based retrieval: sections, tables, and figure captions are the natural retrieval units, because a graph built over arbitrary 500-token windows has nothing clean to anchor its entities to.
Closing
Treat chunking the way you treat schema review: deliberate, documented, and versioned. When retrieval quality disappoints, inspect chunk boundaries before you audition new embedding models, because no ranking function can reunite what splitting separated. And when you do change the strategy, treat re-chunking as a migration with regression checks, not a rerun of a script. The corpus is a database. Model it like one.