Structure-Aware Ingestion, Chunking Was Never the Point
Pull-quote: “The author of that document spent hours organizing it into sections, tables, and headings. Fixed-size chunking throws that work away in milliseconds, then the retrieval team spends a quarter trying to buy it back.”
Why this matters
Most RAG quality problems are blamed on retrieval and caused by ingestion. The 512-token fixed-size chunk with overlap is the standard recipe, and it has a structural flaw: it slices documents by arithmetic, not by meaning. Enterprise answers disproportionately live in exactly the places arithmetic slicing destroys: tables, headed sections, figure captions, definition lists. If the answer was dismembered at ingestion, no reranker downstream can reassemble it.
What fixed-size chunking loses
| Document structure | What holds the answer | What fixed-size chunking does |
|---|---|---|
| Table | Cell + row header + column header + caption | Splits rows from headers; cells become orphan numbers |
| Headed section | Heading gives scope (“Section 7: Exclusions”) | Body text separated from the heading that defines it |
| Figure + caption | The caption interprets the figure | Caption drifts into an adjacent chunk |
| Nested list | Parent item scopes the children | Children retrieved without their parent’s meaning |
| Cross-reference | “as defined in 3.2” | Referenced section not retrieved with the referent |
A retrieved chunk that reads “shall not exceed 40,000. See exclusions above.” is not evidence. It is a fragment with the units, subject, and exclusions all amputated, and the generation model will guess at all three, fluently.
The structure-aware pipeline
PDF / DOCX / HTML
│
▼
Layout-aware parsing ──► headings, paragraphs, tables,
│ figures, captions, reading order
▼
Semantic sectioning ──► units follow the document's own
│ boundaries, not a token counter
▼
Table extraction ──► tables as structured objects +
│ text summary for embedding
▼
Metadata enrichment ──► doc type, section path, dates,
│ entities, effective/superseded
▼
Index: embeddings + lexical + metadata filtersFour design decisions matter more than any chunk size:
- Parse layout, not character streams. A PDF is a rendering, not a text file. Layout-aware parsing recovers reading order, heading hierarchy, table geometry, and captions: the skeleton everything else hangs on.
- Section semantically. Split at the document’s own boundaries (headings, list scopes, table edges) and attach the heading path (
Contract > Section 7 > Exclusions) to every unit. A chunk that knows where it lives is retrievable in a way an anonymous fragment never is. - Extract tables as tables. Store the structured object for exact lookups and a generated text summary for embedding. Embedding raw table text produces vectors that are numerically real and semantically empty.
- Design metadata first. Document type, section path, effective date, superseded-by, named entities. Half of enterprise retrieval quality is filtering (this contract, current revision, that aircraft type), and filters run on metadata that must be captured at ingestion or reconstructed never.
Two operational notes follow. First, regulated corpora often cannot leave the environment they are governed in, so the parsing and structuring stack must be able to run locally; an ingestion pipeline that depends on a cloud parsing API is a non-starter for exactly the documents that need it most. Second, citation quality downstream is a direct function of structure captured at ingestion: a system can only cite the section, page, and table that its pipeline preserved, so the fidelity of every future citation is decided at the door.
Closing
Chunking was never the point; preservation was. The question to ask of an ingestion pipeline is not “what chunk size?” but “what does the retriever see when the answer lives in a table, under a heading, behind a cross-reference?” Parse the layout, section on meaning, keep tables structured, capture metadata at the door. Retrieval can only return what ingestion kept.