Chunking, Validation, JSONL, The Catalog Schema Behind Retrieval
Pull-quote: “A chunk without metadata is a sentence without a speaker. Retrieval can find it, but nothing downstream can decide whether to trust it.”
Why this matters
By the time anyone measures retrieval quality, the decisions that determined it have already been made. They were made at preparation time: how documents were split, what each fragment remembered about its origin, and whether malformed records were caught before embedding. Preparation is usually treated as a throwaway script, which is how corpora end up as folders of anonymous text fragments. The alternative is to treat preparation as data engineering, with the same instrument it always produces: a catalog.
Chunking is a data modeling decision
Splitting documents is unavoidable; embedding models and context windows both impose limits. The mistake is treating the split as a mechanical step. A chunk boundary in the wrong place separates a clause from its condition or a number from its unit, and no downstream cleverness reassembles them. Chunking with live validation, where you see the fragments as they are produced and correct the strategy before committing the corpus, turns an irreversible batch decision into an inspectable one. That inspection loop is a core design element of DocPrep-AI, an open-source, browser-based document preparation tool: real-time chunking and validation, applied across mixed repositories of PDF, DOCX, XLSX, EPUB, HTML, and text, with every byte processed client-side.
What a catalog remembers
The tool lands every record in a 25-column master catalog schema, enriched by AI-assisted classification and tagging. The exact columns matter less than the categories they cover, because each category answers a question someone will eventually ask of the corpus.
| Category | The question it answers | Who asks it |
|---|---|---|
| Identity and provenance | Where did this fragment come from? | Reviewers, auditors, citation rendering |
| Structure | Where did it sit in the source document? | Re-assembly, context expansion |
| Classification and tags | What is it about, what type is it? | Metadata filters at query time |
| Processing lineage | How and when was it produced? | Debugging, re-ingestion, updates |
The retrieval payoff is the third row. Most production retrieval failures are scope failures, the right words from the wrong document type, the current policy shadowed by a superseded draft, and metadata filters are the cheapest fix in the entire stack. Filters are only as good as the catalog behind them. A 25-column record can narrow a query to “policies, current, from this department” before similarity is computed. An anonymous fragment cannot.
Why JSONL at the boundary
Local folder ──► parse ──► chunk ──► validate ──► catalog ──► JSONL
(structure (live (malformed (25 columns, (one record
preserved) feedback) records AI-assisted per line,
caught here) tagging) streamable)
│
embedding pipelines ◄───────┤
vector databases ◄────────┤
CAG workflows ◄────────┘JSONL is the right export format for the same unglamorous reasons it became the standard interchange for ML pipelines: one self-contained record per line, streamable at any corpus size, appendable without rewriting, and diffable line by line. The full catalog record travels inside each line, so every consumer, an embedding pipeline, a vector database loader, a CAG workflow, receives fragments that carry their own provenance instead of consulting a sidecar.
Closing
Retrieval-ready is a property of records, not a property of chunks. Fragments become retrieval-ready when they are validated at creation, cataloged with the metadata that query-time filters and citations will need, and exported in a format any pipeline can stream. That is the whole pipeline, from local folder to 25-column catalog to JSONL, and it is why the projects that treat preparation as data engineering are the ones whose retrieval holds up.