Agentic document processing: what an agent adds after OCR
Pull-quote: “Structured output guarantees the shape of the answer. It guarantees nothing at all about the answer.”
Optical character recognition stopped being the hard part of document automation years ago. Modern engines read a clean scan almost perfectly, and the errors that remain sit where a human also struggles: handwriting, stamps over printed text, a photocopy of a fax. The bottleneck moved to everything after the characters, which is deciding what the text means, forcing it into a schema an operational system will accept, and knowing when the result is wrong.
Agentic document processing puts a model-driven loop in that gap: classify the document, choose an extraction strategy, produce a validated structured record, score how much to trust each field, and route what fails to a person. This post is for the engineer or operations lead who already has an OCR service and cannot work out why the review queue never shrinks. Zorost builds this in aviation and freight, where an action is blocked until a document is read correctly, so the examples come from that work.
By the end you will be able to describe the five stages of a document pipeline, say what structured output does and does not guarantee, set review thresholds per field, and decide whether your documents need an agent at all.
What you will be able to do
- Name the five stages between a scanned page and a posted record, and say which one your pipeline is missing.
- Explain why schema-valid extraction is still frequently wrong, and add the validation layers that catch it.
- Interpret a model-reported confidence score correctly, including what it is not measuring.
- Set per-field review thresholds using the cost of an error rather than one global cutoff.
- Decide between a deterministic parser and an agent using layout variance and volume, not preference.
What does an agent add that OCR does not?
An agent adds judgement between reading and recording: which document this is, which schema applies, whether the values are internally consistent, and whether a person needs to look. OCR returns characters and their positions. Nothing in that output says the number near the top right is the gross weight rather than the tare weight, or whether that weight is plausible given the piece count on line four.
Five stages sit between a page and a posted record. Classification decides what the document is, which is harder than it sounds because one envelope routinely contains three document types. Extraction produces field values against the schema classification selected. Validation checks the record against itself, against reference data, and against arithmetic. Scoring attaches a trust estimate to each field. Routing sends the record to automatic posting, to a targeted field check, or to full review.
A deterministic parser does stages two and three well when layout is fixed. It cannot do stage one across a variable population, and it has nothing useful to say at stage four. The agent’s contribution sits at the edges of the process rather than in the middle, which is why teams that replace a working parser with a model get slower and less accurate at the same time.
Why does extraction break on real documents?
Because production document populations are far more varied than the sample set anyone tests against, and the variance is structural rather than cosmetic. A category like “air waybill” or “bill of lading” is a regulatory concept, not a layout, so one label covers dozens of issuers who each place the same field somewhere different.
- Tables that split across a page boundary, so the last row of page one and the first row of page two are the same line item.
- The same field appearing twice with different values, where one is a correction and the layout does not say which.
- A stamp, signature, or handwritten annotation over a printed field, changing its meaning without changing its printed text.
- Continuation sheets that carry line items but repeat none of the header, so a page read alone has no key.
- Checkboxes and selection marks that carry the document’s entire legal meaning while occupying four pixels.
- Units and formats that vary by origin: kilograms against pounds, comma decimal separators, dates ambiguous below the thirteenth.
- Scans that are rotated, skewed, or a photograph of a screen showing a PDF.
Two of those deserve attention because they produce confidently wrong output rather than obvious failure. The corrected-value case yields a clean, schema-valid record containing the superseded number. The continuation sheet case attaches line items to the wrong parent, which is worse than losing them. Neither is a reading error, so no amount of OCR quality fixes them, and both are invisible to a validator that checks types.
What does structured output actually guarantee?
Structured output guarantees that the model’s response conforms to a schema you supplied. It guarantees nothing about whether the values are correct, and treating conformance as correctness is the most common design error in document pipelines.
The mechanism is worth knowing precisely. With OpenAI’s Structured Outputs you set strict: true on a tool definition or supply a JSON Schema through response_format, and the supported subset carries real constraints: additionalProperties must be false, every property must be listed in required, and an optional field is expressed as a union with null rather than by omission (OpenAI Structured Outputs guide). That last rule has a consequence people trip over. Because the model must emit every key, a field genuinely absent from the document arrives as an explicit null, and so does a field the model could not read. Your schema cannot tell those apart unless you add a separate reason field, so add one.
Four validation layers turn a schema-valid record into a trustworthy one, in this order because each is cheaper than the next.
| Layer | Checks | Example failure it catches |
|---|---|---|
| Type and format | Types, patterns, enumerations, unit presence | A weight parsed as 1,250 becoming 1 rather than 1250 |
| Internal arithmetic | Sums, line totals, piece counts, date ordering | Line items summing to a different total than the printed one |
| Reference data | Codes, identifiers, and parties resolve to real records | An airport or carrier code that does not exist |
| Cross-document | Agreement with a related record already in your systems | A delivery quantity exceeding the ordered quantity |
The third and fourth layers are where an agent earns its place, because both need tool calls rather than parsing: look up the code, fetch the related order, compare. Design those lookups as narrow, read-only tools with tight schemas, the discipline described in tool design for agents.
How do you decide what a human reviews?
Route on the expected cost of the error, per field, rather than on a single document-level threshold. A wrong carrier name is an annoyance. A wrong dangerous-goods classification, part serial number, or airworthiness reference is a different category of problem, and no global cutoff expresses both.
Start by understanding what a confidence score reports, because vendors mean different things. Azure AI Document Intelligence returns a value between zero and one and states it can be used to decide whether to accept a prediction automatically or flag it for review. The important detail sits further down the same page: field-level confidence reflects the model’s confidence in the position of the extracted value, and Microsoft advises combining it with the underlying OCR word confidence (Azure AI Document Intelligence accuracy and confidence). A field can therefore score highly for location and still contain misread characters. The same page documents that merged cells and page-split rows carry lower confidence by design, which is a signal worth routing on directly.
Model-reported confidence from a general language model is weaker still. A token probability describes the model’s own distribution, not the document, and asking a model to rate its certainty produces a number correlated with correctness but poorly calibrated. Treat it as one input, not the decision.
A workable policy has three tiers. Post automatically when every field clears its threshold and all four validation layers pass. Send a targeted field check when one or two fields fail, showing the reviewer only the cropped region and the proposed value. Send the whole document to review when classification was uncertain, a critical field failed, or cross-document validation contradicted an existing record.
Then measure the queue rather than the model. Straight-through rate is the share posted with no human touch. Escalation precision is the share of reviewed documents that actually needed correcting, and it tells you whether your thresholds are too cautious. A pipeline with a ninety percent review rate and a five percent correction rate is a manual process with extra steps.
When does an agent beat a deterministic parser, and when does it lose?
An agent wins when layout variance is high and volume per layout is low. A parser wins when layout is fixed and volume is high, and it keeps winning, because a template tuned against ten thousand documents of the same shape is faster, cheaper, and fully deterministic.
| Condition | Deterministic parser | Agent pipeline |
|---|---|---|
| One layout, high volume | Correct choice. Cheapest per document, testable, no drift | Slower and dearer for no accuracy gain |
| Dozens of layouts, low volume each | Template maintenance never converges | Correct choice. Generalises across issuers |
| New issuer appears weekly | Needs a new template before processing | Handles it on arrival, at lower confidence |
| Semantic judgement required | Cannot express it | Correct choice. Weighs a correction against an original |
| Regulated field, zero error tolerance | Deterministic and auditable | Needs validation and human sign-off regardless |
| Reference lookups and cross-checks | Possible but hand-coded per case | Correct choice. Tool calls generalise |
The honest composition for most operations is both. Route the handful of high-volume, fixed-layout document types to templates, and send the long tail to the agent. That split covers most documents deterministically while removing the template backlog that made the queue unmanageable.
How do you evaluate a document pipeline?
Evaluate per field against a labelled hold-out set, not per document against an impression. Document-level accuracy hides the distribution that matters: a pipeline getting ninety-five percent of fields right can be wrong on almost every document if the failures are spread out, and flawless in production if they concentrate in one unimportant field.
Build the set from real production documents, stratified by issuer and document type, with correct values recorded by the people who currently do the work. Two hundred documents chosen that way beat two thousand from one source. Score exact match for identifiers and codes, normalised match for names and addresses, and numeric tolerance where rounding is legitimate. Keep the per-issuer breakdown, because that tells you a new vendor’s layout is degrading before the queue does.
Track drift explicitly. Populations change without anyone telling you: a carrier redesigns a form, a scanner is replaced, a new trading partner arrives. Re-score the hold-out set on every model or prompt change, and sample recent production documents into it on a schedule. That is the discipline in building an agent evaluation harness, applied to fields instead of trajectories.
Where this goes wrong
Confidence treated as calibrated probability. The symptom is a threshold tuned once that now passes bad records. The cause is assuming 0.9 means nine correct in ten across every field and issuer. The fix is to plot correctness against reported confidence per field and set thresholds from that curve.
Validation that checks the schema and stops. The symptom is clean records that contradict the order they belong to. The cause is treating structured output as a correctness guarantee. The fix is the four-layer table above, with arithmetic and cross-document checks running before anything posts.
The corrected value losing to the original. The symptom is a superseded number posted with full confidence. The cause is an extraction prompt asking for “the gross weight” when the document contains two. The fix is to extract all candidates with their positions, apply an explicit resolution rule, and escalate when it cannot decide.
Continuation sheets processed in isolation. The symptom is line items attached to the wrong parent, or orphaned. The cause is page-level rather than document-level assembly. The fix is to group pages into logical documents first, then check that every line-item page resolves to one header.
A review queue that never shrinks. The symptom is a stable high escalation rate months after launch. The cause is that reviewer corrections go into the operational system and nowhere else. The fix is to write every correction back as a labelled example and change what sits behind the top correction reasons each week.
Everything sent to the model. The symptom is a cost per document that breaks the business case, on documents a template handled for years. The cause is architecture preference beating population analysis. The fix is to measure the volume distribution across layouts and route the head deterministically.
Common questions
Is agentic document processing just OCR with extra steps?
No. OCR converts pixels to characters and positions. Agentic document processing decides what the document is, which schema applies, whether the record is internally and externally consistent, and whether a person should look at it. Reading is one input, and in most pipelines it is no longer the step that fails.
Do I still need OCR if I use a vision language model?
Usually yes, as a parallel signal rather than a replacement. A dedicated OCR engine returns word-level geometry and per-word confidence, which is what lets you crop the exact region for a reviewer and build a composite confidence for a field. Running both gives you a cheap disagreement check: when two readings of a critical field differ, escalate.
What confidence threshold should I start with?
Do not start with a number, start with a labelled set. Score a few hundred real documents, plot correctness against reported confidence for each field separately, and choose the point where the residual error rate is acceptable for that field’s cost of failure. A global threshold is almost always too strict for low-stakes fields and too loose for the two that matter.
Can this run without sending documents to a third-party API?
Yes. Classification, extraction, and validation can all run against a self-hosted model on your own hardware, which is the usual requirement when documents carry commercially sensitive or export-controlled content. The accuracy trade is real at the extraction step, and the way to size it is to run your labelled set against both options before committing.
How do I stop the pipeline from silently getting worse?
Re-score a fixed hold-out set on every prompt, schema, or model change, and refresh that set from recent production traffic on a schedule. Keep the per-issuer breakdown visible, because degradation almost always arrives as one trading partner’s redesigned form rather than a uniform drop.
Next steps
Take one document type, label two hundred real examples by hand, and measure your current pipeline per field. That exercise usually shows two fields cause most of the review load, which turns an open-ended accuracy project into two specific fixes.
Then read getting reliable structured output from a language model for the schema and retry mechanics underneath stage two, and building an agent evaluation harness for the measurement loop that keeps the queue shrinking. If your input problem is tabular rather than documentary, pipeline reliability and the failures that never raise an alert applies the same validation discipline to feeds.
Per-field thresholds and a targeted field check rather than full re-keying is the shape of the document handling Zorost builds in aviation and freight, where the document is what blocks the operational action. We publish what we test in AI Fieldwork.