Multimodal Retrieval and the PDF Problem OCR Never Solved
Pull-quote: “Most of what an enterprise knows lives in PDFs, and much of what those PDFs mean is not in the character stream. OCR reads the characters. The meaning was in the layout.”
Why this matters
The standard ingestion pipeline treats a PDF as a container of text: run OCR or text extraction, get a string, chunk the string. That model is wrong in a way that caps retrieval quality permanently. Documents carry meaning through structure that a character stream cannot represent. A number in a table means nothing without its row and column headers. A figure often contains the finding the surrounding paragraphs only gesture at. Two-column layouts interleave into nonsense when read in raster order. By the time flattened text reaches the chunker, the damage is done, and no embedding model downstream can recover what extraction destroyed.
Where OCR-only pipelines fail
| Content | What OCR alone produces | What retrieval actually needs |
|---|---|---|
| Tables | Row-major word soup; totals detached from labels | Structured rows with headers preserved, table kept atomic |
| Figures and charts | Caption text only, or nothing | Caption plus a description of what the figure shows |
| Multi-column pages | Sentences interleaved across columns | Reading order recovered from layout |
| Scans with stamps and handwriting | Noise, or dropped regions | Region detection; hard regions routed to vision models |
| Forms | Keys and values disassociated | Key-value pairs extracted as fields |
Each row is a class of question the flattened pipeline can never answer correctly, no matter which embedding model ranks the wreckage.
The layered stack
Parse layout first. Segment each page into text blocks, tables, figures, and form regions, and recover reading order before any chunking happens. Then treat each content type on its own terms. Tables get structured extraction: serialize them with headers attached, keep them atomic per the chunking rules from earlier in this series, and store the rows as data so numeric lookups do not have to round-trip through prose. Figures get their captions plus a vision-language model description of what the figure actually shows, indexed as text. And alongside all of it, embed the page images themselves: late-interaction vision retrievers match queries against page images directly, which catches what parsing missed and degrades gracefully on the ugliest scans.
PDF page
│ layout analysis
├─ text blocks ──► reading order ──► text chunks
├─ tables ──► structured rows, headers kept ──► table index
├─ figures ──► caption + VLM description ──► text index
└─ page image ──► vision embedding ──► page-image index
│
all indexes ──► unified retrieval ──► answer cites page + regionProvenance holds the stack together: every retrieved unit carries its page and region, so an answer can cite the exact table on the exact page, and a reviewer can check it in seconds.
Constraints worth designing for
The hardest deployments prove the stack. Quality manuals in regulated plants are decades of scans, stamps, revision marks, and embedded tables, and they often have to be processed air-gapped: parsing, vision models, and retrieval all running locally beside the documents, because the documents do not leave the site. Biomedical literature is the other extreme, where the finding usually lives in a results table or a forest plot; captions and table structure are first-class retrieval units there, not decoration.
Closing
The PDF problem was never an OCR problem. Character accuracy has been good for years; the losses happen in structure, and they happen before retrieval begins. Parse layout first, give every content type its own extraction and its own index, keep page images as the safety net, and carry provenance to the region level. Retrieval quality over documents is decided at ingestion, one page at a time.