Prompting for JSON Is a Probability Game — Constrained Decoding Makes It an Engineering Guarantee
JSON output from LLMs underpins agent tool calls, structured extraction pipelines, and API integrations. A 1.5% schema failure rate on 100,000 calls means 1,500 broken downstream actions; silent semantic failures are worse because nothing alerts. The layered defense described here turns an unreliable probabilistic behavior into a measurable, monitorable engineering property.
Asking an LLM to output valid JSON with a prompt is a probabilistic request, not a system guarantee. A single misplaced token — a trailing comma, a single quote, a hallucinated field name — breaks the entire payload. The failure taxonomy splits into syntax errors that crash parsers, schema violations where legal JSON doesn't match the expected structure, and silent semantic failures where the structure is perfect but the content is wrong.
Constrained decoding changes the game by masking illegal tokens during generation, not after. When the model reaches a field constrained to an enum of three values, it cannot emit a fourth. This, combined with JSON Schema treated as a reliability tool — using enums, descriptions as field-level prompts, shallow nesting, and explicit required fields — shrinks the model's degrees of freedom and reduces downstream breakage.
A production stack layers constrained decoding with Pydantic validators, Instructor-style retry loops, and four-tier validation: syntax, schema, business rules, and semantic correctness. Monitoring schema failure rate, retry rate, and downstream data quality catches drift before it becomes a customer-facing problem. The core shift is from hoping the model behaves to building a system that doesn't let it misbehave.
The article reframes JSON Schema not as a passive data contract but as an active reliability tool that doubles as field-level prompting — descriptions embedded in the schema constrain the model's output space before generation even begins.
Silent failures from hallucinated field names are more dangerous than parse errors because many frameworks ignore unknown fields by default, letting bad data propagate undetected.
Retry rate is positioned as a leading indicator of system drift: a jump from 2% to 14% signals model version changes, prompt rot, or shifting input distributions before business metrics degrade.
The four-layer validation model (syntax → schema → business → semantic) makes explicit that JSON Schema enforcement only answers 'does this look right,' never 'is this actually correct' — the hardest layer requires ground truth, cross-checks, or human review.
Splitting deeply nested schemas into multiple LLM calls is presented as a practical reliability tactic, not just a performance optimization — each call has a smaller output space and fewer token decisions that can compound into failure.