Structured output: getting data instead of prose from a model

Pull-quote: “Constraint removes options, it never adds knowledge. A schema can force a well-formed date and still receive an invented one.”

Structured output is how you get data instead of prose out of a model, and the mechanism you choose trades reliability against reasoning quality. There are five distinct mechanisms, they give genuinely different guarantees, and the strongest ones can make the answer worse if you apply them at the wrong moment.

This post is for engineers who parse model output in production code. It follows tool design for agents, because tool calling is one of the five mechanisms and the schema discipline is the same in both places.

By the end you will be able to pick a mechanism for a given task, design a schema models comply with, validate and repair without leaking cost, stream partial structures safely, and recognise when the schema is the thing hurting your answers.

What you will be able to do

• Choose between prompt-and-parse, JSON mode, tool calling, grammar constraints, and schema-constrained sampling.
• State what each mechanism guarantees, and stop assuming any of them guarantee correct content.
• Write schemas that models comply with: flat, enumerated, required, and described.
• Build a validate-and-repair loop that is bounded and observable rather than a cost leak.
• Stream a partial structure for perceived latency without acting on an unfinished value.
• Recognise when a schema is suppressing reasoning, and when to skip structure entirely.

What are the five ways to get structured output from an LLM?

There are five mechanisms, ordered from weakest guarantee to strongest: prompt and parse, JSON mode, function or tool calling, constrained decoding with a grammar, and schema-constrained sampling. They are not interchangeable, and the difference is what happens when the model is wrong.

Prompt and parse. You ask for JSON in the prompt and parse whatever comes back. The guarantee is nothing at all: expect fenced code blocks, a sentence before the opening brace, and trailing commentary. It works with every model and every endpoint, including completion-only APIs, which is why it survives. Budget for defensive code that extracts the JSON substring, parses tolerantly, and validates.

JSON mode. A request-level option that constrains the response to syntactically valid JSON, available on most hosted chat APIs and several self-hosted servers. The guarantee is that it parses, not that it matches your schema, so invented field names and a valid but empty object are both possible. A response cut off at the token limit is also invalid JSON, so check the finish reason before trusting the parse.

Function or tool calling. You declare your schema as a tool and treat the emitted call as your payload. Field-name fidelity is substantially better than asking in prose, several providers offer a strict mode that validates arguments against the schema before returning, and the schema keywords supported in strict mode differ by provider [NEEDS SOURCE]. It also reuses the code path you already have for tools, and the model can decline to call, which is a signal worth reading.

Constrained decoding with a grammar or regex. At each sampling step the decoder is restricted to tokens that keep the output on a legal path through a formal grammar or a regular expression. The guarantee is syntactic validity by construction, because an illegal token cannot be sampled. Self-hosted servers are where you find it, with llama.cpp accepting a GBNF grammar and servers such as vLLM exposing guided decoding. It costs grammar compilation and a small per-token overhead, and grammars get awkward for recursive structures.

Schema-constrained sampling. The same decoding mechanism, except you supply a JSON Schema and the server compiles it into the grammar. The guarantee is conformance to the supported subset of your schema, and that subset varies between implementations, with unsupported keywords ignored rather than enforced [NEEDS SOURCE]. Test your schema against your own server rather than assuming coverage.

None of the five guarantees content. Constraint removes options, it never adds knowledge, so a schema can force a well-formed date string and still receive a date that appears nowhere in your source. Shape is a syntax property and truth is not, which is why validation has two layers rather than one.

How do you design a schema a model will comply with?

Models comply with schemas that are flat, small, and named in the language of the task. Compliance is a property of your schema design at least as much as of the mechanism enforcing it.

• Flat beats nested. Prefer a top-level array of records to a tree, because omissions cluster inside deep optional branches.
• Short enums in the model’s own vocabulary. ["low", "medium", "high"] outperforms ["S1", "S2", "S3"] unless you explain the codes, and you can map to codes after validation.
• Field names are instructions. evidence_quote_verbatim tells the model more than text2 ever will.
• One imperative sentence per field description. Say what goes in the field and what does not.
• Avoid unions. A union asks for a type decision the model makes inconsistently. Use a discriminator enum or split into two calls.
• Require fields and use sentinels. An optional field invites omission, while a required field holding "unknown" tells you the model considered it.
• One example, not three. Extra examples cost context on every request and encourage copying their values.
• Leave a place for the leftovers. Without a notes or unmapped field, whatever does not fit gets forced into a field where it does not belong.

{
  "name": "extract_findings",
  "schema": {
    "type": "object",
    "properties": {
      "findings": {
        "type": "array",
        "maxItems": 20,
        "items": {
          "type": "object",
          "properties": {
            "claim": { "type": "string", "maxLength": 240, "description": "One sentence in your own words." },
            "evidence_quote_verbatim": { "type": "string", "maxLength": 400, "description": "Exact text copied from the source. Never paraphrase." },
            "source_page": { "type": "integer", "minimum": 1 },
            "status": { "type": "string", "enum": ["supported", "contradicted", "not_addressed"] }
          },
          "required": ["claim", "evidence_quote_verbatim", "source_page", "status"],
          "additionalProperties": false
        }
      },
      "unmapped_notes": { "type": "string", "description": "Relevant material that fits no finding. Empty string if none." }
    },
    "required": ["findings", "unmapped_notes"],
    "additionalProperties": false
  }
}

Two details there do real work. not_addressed gives the model a legal way to say the source is silent, and maxItems stops a long document pushing you into a truncated array. A schema is also context, so it competes with your evidence for the window, which is the accounting in treating the context window as a budget.

How should you validate and repair a malformed response?

Validate with a real schema validator, allow exactly one repair attempt that includes the validator’s error message, then fail loudly. A JSON Schema validator, a Pydantic model, or a Zod schema all work; hand-rolled field checks miss the cases that actually break you.

Validation has two layers. The syntactic layer asks whether the payload parses and matches types, enums, and required fields. The semantic layer asks whether the values are true to your inputs: dates inside the document’s range, identifiers that exist, totals that reconcile. Constrained decoding satisfies the first layer completely and tells you nothing about the second.

For extraction work, one semantic check is worth more than the rest combined: assert that evidence_quote_verbatim is a substring of the source text. It catches fabrication at the field level, deterministically, for the cost of a string search.

The repair turn carries three things: the invalid output, the validator’s error message, and an instruction to return only the corrected object. Leave the source material out where you can, so the repair costs a fraction of the original call.

Bound the loop at one attempt. Every attempt pays for the prompt again, so an unbounded repair loop is both a cost leak and a latency cliff, covered as a spend pattern in cost and latency engineering for LLM systems. A model that fails the same schema twice is usually telling you the schema is wrong, the task is too large, or the input does not contain what you asked for. Escalate to a simpler schema, a split task, a different model, or a person.

You know the loop is instrumented when your logs show repair_attempt=1 and never a 2, and structured_output_failed is a counted outcome carrying the schema name and the failing field. Repair rate per field tells you which field to redesign.

How do you stream a partial structure without breaking the parser?

Stream for perceived latency by parsing the buffer incrementally with a tolerant parser, and act only on values that are complete. On each chunk, attempt a partial parse that closes any open string, array, and object, which yields a provisional value you can render.

Order your fields for streaming, because they arrive roughly in schema order. Put the summary field first so the user has something to read while an array fills behind it. Arrays are where streaming genuinely pays, since each element can render as it closes.

Two rules keep this safe. Render partials, but never act on them: a half-streamed argument is not a decision the model has finished making. And validate the final buffer even though you rendered along the way, because everything displayed was provisional.

Truncation is the failure that catches people. A stream that ends because it hit the token limit leaves a genuinely incomplete object, and your partial parser will cheerfully close the braces and hand you something that looks complete. Read the finish reason on every response and treat a length stop as a failure rather than a value.

Why does a strict schema sometimes make the answers worse?

Forcing a schema too early suppresses the intermediate reasoning the model needs, because constrained sampling restricts the token space at every step. If the first token must be {, there are no tokens left in which to work the problem out, and whatever those tokens would have contributed is simply gone.

A narrow enum causes the same damage in a smaller space. Ask for ["supported", "contradicted"] on a source that addresses neither, and you have designed a schema that requires a guess, then measured the guess as an answer. Every enum needs a legal escape value such as not_addressed, unknown, or insufficient_evidence.

The pattern that works is reason first, emit second. In one call, make a reasoning string the first required field in the schema so the model has room before it commits. In two calls, let a cheap unconstrained call do the thinking and a constrained call extract the structure from that output. Two calls cost more tokens and are easier to evaluate, cache, and debug, which is usually the better trade for anything you have to defend.

Two cautions. A reasoning field is working material, so do not display it as a conclusion or store it as a fact. And do not over-constrain content either, since a maxLength of 40 on a claim field produces truncation-shaped answers rather than short ones.

When should you not use structured output at all?

Skip structured output when the consumer is a person reading prose, when you cannot yet name the fields, or when a deterministic parser would do the job better. Reaching for a schema by reflex costs you quality in all three cases.

Prose for humans should stream as markdown, because routing it through JSON adds escaping problems and a parse step in exchange for nothing. Exploratory analysis should run unconstrained first, since a schema forces the output into the fields you guessed and you never see what you missed; design the schema from what the open pass returned. A five-label classification needs a label, not an object. And a fixed-layout input such as a form or a table belongs to a parser, which is cheaper and more reliable than any model.

Where this goes wrong

• Valid JSON, wrong shape. The payload parses, then a field is missing or renamed, because JSON mode was doing the work and nothing checked the schema. Validate properly, or move to tool calling or schema-constrained sampling.
• Truncated object accepted. An array silently loses its tail on long inputs, because the response hit the token limit and the tolerant parser closed it. Check the finish reason, cap maxItems, and chunk the input.
• Repair loop cost leak. A few requests show latency spikes and outsized token spend, because validation failures retry without a bound. Cap at one repair, count failures, and escalate.
• Reasoning suppressed by the schema. Output complies perfectly and the values are shallow, because constrained decoding started at the first token. Reason in an unconstrained field or call, then emit.
• Forced choice with no escape. Confident labels appear on inputs that support no label, because the enum had no unknown member. Add the escape value and treat its rate as a data quality signal.
• Fabricated quotes passing validation. A verbatim field holds text that is not in the document, because type validation only checked that it was a string. Assert the substring and reject the record.

Common questions

Is JSON mode enough on its own?

No. JSON mode typically guarantees that the response parses, not that it matches your schema, so you can get valid JSON with renamed fields or an empty object. Treat it as a parsing convenience and keep a schema validator behind it.

Should I use tool calling even when I do not want an action?

Often, yes. Declaring your schema as a tool and treating the emitted call as a payload reuses the code path you already have and tends to produce better field-name fidelity than asking in prose. It also lets the model decline, which is more useful than an empty object.

Does constrained decoding make a model worse at reasoning?

It can, if it starts too early. Constrained decoding removes token options and adds no knowledge, so committing to a schema at the first token takes away the space the model would have used on the problem. Reason first in an unconstrained step, then constrain the emitting step.

JSON or YAML for structured output?

JSON. It has fewer ambiguities, better validator support, and no whitespace significance, which matters because a model that misindents YAML produces an error it cannot easily localise. For streaming records, use JSON Lines rather than one large document.

What about outputs too large for one response?

Do not ask for one giant object. Chunk the input, request one record or one small array per chunk, and merge in your own code where the merge is deterministic and testable. That also keeps any single failure small enough to repair cheaply.

Next steps

1. Add two things to your extraction path: a real schema validator and a counted structured_output_failed outcome. Until failures are counted per field, you are guessing about which field to redesign.
2. Continue the course with evaluating agents by grading runs. Schema compliance is easy to measure and easy to over-trust, and an evaluation harness is where you learn whether the values inside the shape were right.
3. Before tightening a schema, check the failure it is meant to prevent against the failure taxonomy for production agents, since several of these show up as silent truncation rather than parse errors.
4. This is the extraction discipline behind EvidAI in pharmaceutical evidence review, where every extracted claim carries a verbatim quote that is checked against the source document before a reviewer ever sees it.