Getting Reliable JSON Out of an Agent (Schemas, Constrained Decoding, Retries)

A pipeline I built last year fell over at 2 a.m. because a model decided to wrap its JSON in a chatty preamble: "Sure! Here's the data you asked for:" followed by a code fence. My parser, which expected the response body to start with {, threw. The retry hit the same friendly model, got the same friendly preamble, and threw again. Three retries, three failures, one paged human.

The annoying part is that the JSON inside the code fence was perfect. The model did the hard part — extraction — correctly. It just couldn't resist saying hello first. I'd spent my effort on the prompt and none of it on the contract.

That's the whole problem in one anecdote. The model is usually capable of producing the data you want. The failure is in the seams: the preamble, the trailing comma, the field it renamed, the enum value it invented, the response that got cut off mid-object because you forgot to raise the token limit. None of those are reasoning failures. They're contract failures, and you defend against them with mechanism, not better prompting.

The short version: don't ask for JSON, constrain the model to produce it. Use your provider's schema-constrained structured output mode (now standard across OpenAI, Anthropic, and Google), validate the result against a schema you own, and keep a repair-retry loop for the cases that still slip through. Below is the ladder from weakest to strongest, and where I actually stop climbing.

The ladder, weakest to strongest

There are roughly five rungs, and most people sit one rung lower than they think.

Rung one: prompt-and-pray. You write "respond with JSON only, no other text" and hope. This works often enough in a demo to be dangerous. It fails on preambles, code fences, trailing prose, and any model that's having an expressive day. Don't ship this.

Rung two: JSON mode. A provider flag that guarantees the output is syntactically valid JSON — it'll parse. It does not guarantee the output matches your shape. OpenAI's docs are blunt that this is the older, weaker option: "Structured Outputs is the evolution of JSON mode. While both ensure valid JSON is produced, only Structured Outputs ensure schema adherence" (OpenAI structured outputs guide). JSON mode stops the preamble problem. It does nothing about a missing field or a hallucinated enum.

Rung three: schema-constrained structured outputs. You hand the provider a JSON Schema and it constrains generation so the output conforms — every required field present, every type correct, every enum value drawn from your list. This is the rung to live on for data extraction.

Rung four: tool / function calling. The model emits a call to a function whose parameters are a JSON Schema. Mechanically this is the same constraint as rung three, applied to tool arguments instead of a response body. You want it when the model is choosing which action to take, not just filling a form.

Rung five: constrained / grammar-based decoding you run yourself. With open-weight models you control the decoder, so you can mask the token logits at every step to forbid any token that would break the grammar. This is the strongest guarantee — the model literally cannot emit an invalid token — and the most work.

Most production code should sit on rung three or four using a hosted provider. Let me show what each looks like.

Rung three: schema-constrained structured output

All three major providers now ship this, and the shape is similar enough to learn once.

OpenAI uses response_format with a json_schema and strict: true. The strict flag is what turns "please" into "guaranteed." Its requirements are specific and easy to trip over: additionalProperties must be false on every object, and every property must appear in the required array. Optional fields are expressed as a union with null, not by omission (OpenAI structured outputs guide).

Anthropic added the same capability to the Claude API — now generally available, with the beta header (structured-outputs-2025-11-13) no longer required. You pass an output_config.format for a JSON response body, and you set strict: true on tool definitions for guaranteed-valid tool inputs. The schema subset is the same flavor as OpenAI's: additionalProperties: false on every object, every property listed in required (Anthropic structured outputs docs).

Google's Gemini does it through the generation config: set responseMimeType to application/json and pass your schema as responseSchema. Their SDKs let you define the schema with Pydantic in Python or Zod in TypeScript instead of hand-writing JSON Schema (Gemini structured output docs).

Here's the TypeScript shape with OpenAI. I define the schema in Zod and convert it, because hand-maintaining JSON Schema is how the schema and your types drift apart.

import OpenAI from "openai";
import * as z from "zod";

const Ticket = z.object({
  priority: z.enum(["low", "medium", "high", "urgent"]),
  category: z.enum(["billing", "bug", "feature", "other"]),
  summary: z.string().max(280),
  needs_human: z.boolean(),
});

const client = new OpenAI();

const res = await client.chat.completions.create({
  model: "gpt-4o-2024-08-06",
  messages: [
    { role: "system", content: "Classify the support ticket." },
    { role: "user", content: ticketText },
  ],
  response_format: {
    type: "json_schema",
    json_schema: {
      name: "ticket",
      strict: true,
      schema: z.toJSONSchema(Ticket),
    },
  },
});

const parsed = Ticket.parse(JSON.parse(res.choices[0].message.content));

z.toJSONSchema() is built into Zod 4 and targets Draft 2020-12 by default (Zod JSON Schema docs). Note the priority and category fields are enums — with strict mode on, the model cannot return "critical" even if it wants to, because that token is not in the allowed set. That single property kills a whole class of bug.

One thing the constraint does not save you from: it guarantees the shape, not the meaning. A constrained model will happily put a real category on a ticket it misread. So I still run Ticket.parse() at the end — partly against schema drift between my types and the wire format, partly because I sometimes add semantic checks that JSON Schema can't express.

Rung four: tool calling, and when to prefer it

Tool calling is the same constraint pointed at a different target. Instead of "fill this response body," it's "if you call a tool, your arguments must match its schema." Anthropic's strict tool use makes that guarantee explicit: with strict: true, "Claude's tool inputs exactly match your schema" (Anthropic structured outputs docs).

The reason to reach for tools over a plain structured response is choice. When the model picks among search_orders, issue_refund, and escalate, each with its own argument shape, you want function calling — the discriminated union of which tool, plus validated args for that tool, is exactly the agent-loop primitive. When you just need one fixed shape back from one call, a structured response body is simpler and you skip the tool-call ceremony.

A practical TypeScript-ish sketch of the loop:

// model returns tool_calls; each has a name + JSON arguments
for (const call of message.tool_calls ?? []) {
  const tool = tools[call.function.name];
  if (!tool) throw new Error(`hallucinated tool: ${call.function.name}`);

  // strict mode guarantees this parses to the tool's shape,
  // but validate anyway — providers differ, and you own the contract
  const args = tool.schema.parse(JSON.parse(call.function.arguments));
  const result = await tool.run(args);
  // feed result back into the conversation, continue the loop
}

I keep the hallucinated tool guard even with strict mode, because the set of which tools exist and the set of valid arguments per tool are enforced by different parts of the stack, and I don't want to assume both are airtight on every provider.

Rung five: constrained decoding you run yourself

When you run an open-weight model, you own the decoder, which means you can enforce grammar at the token level: at each step, mask out any token that would make the partial output un-parseable against your schema. The model cannot produce invalid JSON because invalid tokens are never on the menu.

The library most people start with is Outlines, which supports regex, JSON Schema, and full context-free grammars in EBNF. It's a real capability with a real cost: the early finite-state-machine approach can spend a long time compiling complex schemas, and a JSON-schema benchmark found compilation timeouts dragged its compliance rate down on heavy schemas. The backend most high-throughput serving stacks moved to is XGrammar — the default structured-generation backend in engines like vLLM and SGLang, with very low per-token overhead (XGrammar). If you're self-hosting and structured output is on the hot path, that's where the ecosystem went.

One caveat worth knowing before you reach for this: hard grammar constraints can interact badly with reasoning. There's published work arguing that forcing structure too early can hurt the model's actual answer quality versus letting it think in prose and structuring afterward — the so-called alignment tax of constrained decoding (arXiv). The practical read: constrain the output, but give the model room to reason first if the task is hard. Don't force the model to think in JSON.

The failure modes, and how to actually defend

Constraint handles shape. These are the things it doesn't, and they're what page you.

Truncation. The single most common one. The model hits the output token limit mid-object and you get half a JSON document that no constraint can rescue — it was valid right up until it stopped. Defenses: set max_tokens generously for the schema's worst case, and on a parse failure check whether the response stopped because of length before you blindly retry. Retrying a truncation with the same limit just burns money.

Schema drift. Your code's expected shape and the schema you sent the model fall out of sync — usually because someone hand-edited the JSON Schema. The fix is to never hand-write it: derive the JSON Schema from the same Zod or Pydantic type you parse with, so there's one source of truth. Instructor leans on exactly this, wrapping the client so a Pydantic model defines the schema, validates the response, and retries on failure in one object.

Hallucinated enum values. A model under prompt-and-pray invents "critical" when your enum is low|medium|high. Strict structured output kills this outright at rung three — the invalid token is unreachable. If you're not on a constrained mode, a validating parser turns it into a catchable error instead of a downstream surprise.

Over-nesting and silent renames. The model returns the right data under customer_name when you asked for name, or wraps everything in an extra { "result": ... } layer. additionalProperties: false plus an exhaustive required list is what forbids the model from adding or renaming fields; this is precisely why both OpenAI and Anthropic make those two constraints mandatory for strict mode.

The repair loop ties it together. When validation fails, don't just retry the same call — feed the error back:

import instructor
from pydantic import BaseModel

client = instructor.from_provider("openai/gpt-4o")

class Ticket(BaseModel):
    priority: str
    summary: str

# on a validation error, Instructor sends the Pydantic error message
# back to the model and asks it to fix the specific field — then re-validates
ticket = client.chat.completions.create(
    response_model=Ticket,
    max_retries=2,
    messages=[{"role": "user", "content": ticket_text}],
)

Showing the model the specific validation error ("priority must be one of low, medium, high") is far more effective than a blind retry, because you've told it exactly what to fix. But cap the retries — two is plenty. A model that fails validation three times in a row is usually failing for a reason more retries won't solve, and you want to degrade gracefully: return a typed error, fall back to a default, or route to a human. Looping forever on a poison input is how you get my 2 a.m. page.

What I actually reach for, by use case

For data extraction or classification against a hosted model — the common case — use your provider's strict schema-constrained mode, derive the schema from Zod or Pydantic, and validate the result with the same type. That's rungs three plus a validating parser, and it covers most of what people are building.

For agents that choose actions, use strict tool calling. The which-tool-plus-validated-args shape is the thing you want, and the strict flag closes the argument-validation gap. Keep a guard for tool names you don't recognize.

For self-hosted open-weight models on a hot path, use grammar-constrained decoding through your serving engine — XGrammar-class backends are fast enough now that there's little reason to hand-roll it. Let the model reason in prose first if the task is hard, then constrain the structured part.

And regardless of rung: validate at the boundary with a schema you own, and keep a bounded repair loop with graceful degradation. The constraint stops the model from emitting garbage. The validator stops your assumptions from being the garbage. Both my pipeline and my sleep got better when I stopped trusting the prompt and started enforcing the contract.