Prompting for JSON Is a Probability Game — Constrained Decoding Makes It an Engineering Guarantee
Why Relying Only on Prompts to Make LLMs Output JSON Is Unreliable
Production-grade structured output from LLMs should not depend on "model voluntarily follows the format," but should instead seal off uncertainty layer by layer through "generation-stage constraints + Schema design + application-layer validation + Retry + monitoring."
1. Why Relying Only on Prompts to Make LLMs Output JSON Is Unreliable
Suppose you write:
Please strictly output JSON, do not output any explanation.
It looks very clear, but to an LLM, this is still just a natural language instruction.
The model is essentially doing:
Based on the previous tokens
predict the probability distribution of the next token
↓
select a token
↓
continue predicting
It does not have a natural "JSON compiler" inside checking whether:
{
"name": "Alice",
"age": 20
}
is strictly legal.
So the model might output:
Sure! Here is the JSON:
{
"name": "Alice",
"age": 20,
}
A human knows what it means at a glance, but program execution:
json.loads(response)
will fail directly.
A very important way of thinking:
JSON correctness is not a single judgment, but the combined result of many token decisions.
In a long JSON, as long as one position generates incorrectly:
"
,
}
[
null
true
the entire result can be invalidated. The article therefore treats "relying only on Prompts" as a probabilistic guarantee, not an engineering guarantee. So:
Prompt:
"Please output valid JSON"
is essentially:
the model "tries its best" to comply
rather than:
the system "guarantees" it complies
These two are a very big difference in production systems.
2. Three Typical Failure Categories
Breaking failures into three layers is very worth remembering:
Syntax
↓
Schema
↓
Semantics
First look at the first two layers and Hallucinated Structure.
Syntax Failure: The grammar is broken
For example:
{
'name': 'Alice'
}
This is Python dict style, not legal JSON, because JSON must use double quotes.
Or:
{
"name": "Alice",
}
Trailing comma.
Or:
Here is your JSON:
{
"name": "Alice"
}
If your parser expects the entire response to be JSON, it will also fail.
So the problem with Syntax Failure is:
Cannot even pass the JSON parser.
That is:
json.loads(...)
directly throws an error.
Common problems include mixed quotes, trailing commas, unquoted keys, explanatory text before JSON, and output truncated by token limit, etc.
Schema Compliance Failure: JSON is legal, but the structure does not meet requirements
For example, you require:
{
"user_id": 123,
"tags": ["ai", "llm"]
}
Schema:
{
"type": "object",
"properties": {
"user_id": {
"type": "integer"
},
"tags": {
"type": "array"
}
},
"required": ["user_id", "tags"]
}
LLM returns:
{
"user_id": "123",
"tags": "ai"
}
This JSON:
Syntax ✅
JSON parse ✅
Schema ❌
The problem is:
user_id should be integer
but returned string
tags should be array
but returned string
Or directly missing a field:
{
"user_id": 123
}
Still:
JSON legal ✅
Schema ❌
Especially for deeply nested Schemas, it becomes increasingly difficult for the model to maintain structural consistency, so excessive nesting significantly increases the error probability.
Hallucinated Structure: The structure "looks very reasonable," but it is not your structure
This is a very dangerous category.
You require:
{
"analysis": "...",
"score": 0.9
}
Model outputs:
{
"analysis_result": "...",
"score": 0.9
}
A human looks at it:
analysis_result and analysis are pretty much the same, right.
But code:
result["analysis"]
may fail directly.
More troublesome is that certain languages/frameworks will:
ignore unknown fields
so the system continues running.
Finally manifesting as:
LLM call successful
JSON parse successful
Program did not error
but data was lost
This is what is called:
silent failure.
This type of problem is especially dangerous because it does not explode immediately, but may only appear several steps downstream.
3. How Constrained Decoding Actually Works
This is one of the most important technical concepts in the entire article.
Normal generation:
LLM
↓
predicts probabilities of all tokens
"{" 0.15
"Hello" 0.10
"Sure" 0.08
"[" 0.07
"name" 0.05
...
The model can theoretically choose any token.
Constrained Decoding adds a:
Grammar / Schema Filter
becoming:
LLM probabilities
↓
Schema / Grammar check
↓
set illegal token probability to 0
↓
select only from legal tokens
For example, now already generated:
{
"sentiment":
Schema is:
{
"sentiment": {
"enum": [
"positive",
"negative",
"neutral"
]
}
}
At this moment:
"positive" ✅
"negative" ✅
"neutral" ✅
"banana" ❌
123 ❌
{ ❌
Then the decoder will directly mask illegal choices.
Conceptually similar to:
for token in vocabulary:
if grammar.allows(token):
keep_probability(token)
else:
probability[token] = 0
Then re-normalize probabilities:
legal tokens
↓
continue sampling
So the most critical point is:
It does not check after generation is complete, but does not allow errors to occur during generation.
OpenAI's public description of Structured Outputs also explicitly describes this dynamic constrained decoding: the system continuously calculates the next allowed tokens based on "already generated tokens + Schema" and masks illegal tokens.
It can be understood as:
Prompt only
LLM
↓
"please follow the grammar"
vs
Constrained decoding
LLM
↓
grammar gatekeeper
↓
illegal tokens simply cannot come out
Therefore reliability is completely different.
4. The Difference Between JSON Mode and Structured Outputs
This is the place most easily confused.
JSON Mode
What JSON Mode guarantees is mainly:
output is valid JSON
For example:
{
"foo": "bar"
}
But you originally required:
{
"sentiment": "positive",
"confidence": 0.95
}
The model may still give:
{
"foo": "bar"
}
JSON is legal, so:
JSON Mode:
Syntax ✅
Schema ???
OpenAI's official documentation currently still explicitly states: JSON mode ensures valid JSON, but does not guarantee matching a specific Schema.
Structured Outputs
The goal of Structured Outputs is:
JSON legal
+
conforms to specified Schema
For example Schema:
{
"type": "object",
"properties": {
"sentiment": {
"type": "string",
"enum": [
"positive",
"negative",
"neutral"
]
},
"confidence": {
"type": "number"
}
},
"required": [
"sentiment",
"confidence"
],
"additionalProperties": false
}
Then structural constraints will forbid:
{
"foo": "bar"
}
Also forbid:
{
"sentiment": 123
}
And:
{
"sentiment": "happy"
}
So it can be remembered as:
| Mode | JSON legal | Schema compliant |
|---|---|---|
| Prompt only | Not guaranteed | Not guaranteed |
| JSON Mode | ✅ | ❌ Not guaranteed |
| Structured Outputs | ✅ | ✅ |
This is one of the most important distinctions in the entire structured output system.
5. Why Schema Design Itself Is Reliability Engineering
Many people think:
Schema = data format
Actually for LLMs it is more accurate:
Schema
=
data format
+
model hint
+
output space definition
+
part of business rules
Give an example.
Schema A:
sentiment: str
Actually allows:
positive
negative
neutral
happy
angry
good
bad
mixed
unclear
...
The model's selection space is very large.
Schema B:
sentiment: Literal[
"positive",
"negative",
"neutral"
]
Selection space directly shrinks to:
3
Going further:
sentiment: Literal[
"positive",
"negative",
"neutral"
] = Field(
description=
"Customer sentiment based only on explicit tone."
)
Schema even begins to take on the role of Prompt.
So:
Good Schema
↓
reduces model degrees of freedom
↓
reduces ambiguity
↓
reduces retry
↓
reduces edge errors
↓
improves downstream reliability
Schema Design is Reliability Engineering.
Google's current Structured Outputs documentation also explicitly recommends using clear description, strong typing, and enum, while still validating values at the application layer.
6. How to Design description, required, and Nesting Levels
description
Don't just write:
sentiment: str
Better:
sentiment: str = Field(
description=
"Customer sentiment based on explicit wording "
"in the message, not inferred intent."
)
Because the Schema is usually seen by the model.
Thus:
description
≈
field-level Prompt
Call it a kind of "Prompt Engineering embedded in the type system."
required
Suppose:
"properties": {
"email": {"type": "string"}
}
But without:
"required": ["email"]
The model may think:
don't know email
→ then I just won't output it
If business logic actually depends on email:
send_email(result.email)
The problem is postponed to downstream.
So important business fields should be explicitly required.
Limit nesting
Not recommended:
customer
└── profile
└── preferences
└── communication
└── email
This kind of 5-layer structure.
The article suggests usually controlling to:
2–3 layers
If continuously appearing:
A
└ B
└ C
└ D
└ E
usually should consider:
flatten
or:
split into multiple LLM calls
For example:
{
"customer_name": "...",
"preferred_language": "...",
"email_opt_in": true
}
is easier to generate, validate, and debug than complex tree structures.
7. Pydantic Validators + Instructor + Retry
Now entering the second line of defense.
Structured Outputs can achieve:
structurally correct
but not necessarily:
business correct
So you can define:
class ExtractedData(BaseModel):
entity_name: str
confidence: float
Then Validator:
@field_validator("confidence")
def valid_confidence(cls, v):
if not 0 <= v <= 1:
raise ValueError(
"confidence must be between 0 and 1"
)
return v
LLM returns:
{
"entity_name": "OpenAI",
"confidence": 4.7
}
Structurally:
JSON ✅
Schema ✅
Because:
confidence = number
But business rule:
0 <= confidence <= 1
not satisfied.
So Pydantic:
ValidationError
Instructor can then feed this error back to the LLM:
Your previous response failed validation:
confidence must be between 0 and 1.
Please correct the response.
Model second time:
{
"entity_name": "OpenAI",
"confidence": 0.92
}
Thus:
Pass.
Complete flow:
LLM
↓
Structured Output
↓
Pydantic
↓
Validator
↓
PASS ─────────→ downstream
FAIL
↓
validation error
↓
Instructor
↓
Retry LLM
↓
re-validate
This is what is called:
Validation → Feedback → Retry Loop.
Retry should be limited, generally serving as a mechanism for handling long-tail errors, not using retry to cover up bad Prompts or bad Schemas.
8. Why structural correctness ≠ semantic correctness
This is the most important layer.
For example input:
This product is absolute garbage, I will never buy it again.
Output:
{
"sentiment": "positive"
}
Schema:
{
"sentiment": {
"type": "string",
"enum": [
"positive",
"negative",
"neutral"
]
}
}
Check result:
JSON legal ✅
Schema Validation ✅
Semantic Correctness ❌
Why?
Schema only knows:
positive is one of the allowed values
It does not know:
the original text expresses anger/negative emotion
That is to say:
Schema:
"Can this answer look like this?"
Semantic validation:
"Is this answer actually correct?"
These are two completely different questions.
9. The Boundary Between Schema Validation and Semantic Validation
The system can be understood as four layers.
Layer 1
Syntax validation
Is it legal JSON?
For example:
json.loads(...)
↓
Layer 2
Schema validation
Are fields, types, enum correct?
For example:
Pydantic
JSON Schema
Zod
↓
Layer 3
Business validation
Are business constraints satisfied?
For example:
confidence ∈ [0,1]
end_date >= start_date
price >= 0
currency must match account
↓
Layer 4
Semantic validation
Is the content consistent with the original information?
For example:
Original text:
"This product is absolute garbage"
sentiment:
positive
→ semantic failure
The last layer is the hardest.
It may require:
rules
ground truth
another model
cross-check
RAG
database
human review
to confirm.
Therefore:
JSON Schema
cannot prove
the LLM is right
it can only prove
the LLM's answer "looks compliant"
Even with Schema enforcement, application-layer validation is still necessary, because structural constraints cannot capture all semantic errors.
Google's official documentation similarly reminds that even if Structured Outputs produce format-compliant JSON, you should continue to validate schema-compliant but semantically wrong values at the application layer.
10. What Should Be Monitored in Production
The article focuses on giving three metrics.
Schema validation failure rate
Definition can be understood as:
Number of requests where first output Schema validation failed
──────────────────────────────
Total LLM requests
For example:
100,000 calls
1,500 schema failures
= 1.5%
This metric mainly reflects:
structural reliability
If there are still a large number of failures after using strict structured output, you should investigate Schema, model invocation method, truncation, provider limitations, etc.
Retry Rate
For example:
100,000 calls
12,000 retried at least once
Retry rate = 12%
This is usually a very good "system health" signal.
Suppose originally:
2%
Suddenly:
14%
You need to investigate:
Model version updated?
Prompt updated?
Schema updated?
User input distribution changed?
New languages entering traffic?
Long inputs increased?
Special emphasis:
Retry rate rising is often an early signal of drift.
Downstream Data Quality
This is the most easily overlooked.
Because:
Schema Validation
can only capture structural errors
What is truly dangerous is:
structure completely correct
but content wrong
For example credit review:
{
"risk": "low",
"reason": "..."
}
Structure perfect.
But the correct answer is actually:
risk = high
This kind of error can only be discovered through:
downstream anomalies
human review
business metrics
ground truth
So mature systems monitor:
LLM boundary metrics
+
business outcome metrics
and not just:
API error rate
It is recommended to alert on the first two metrics, and do continuous sampling and anomaly analysis on downstream quality.
11. How to Choose Tech Stack for Self-Hosted Models vs Cloud APIs
This part can be summarized into two typical architectures.
Self-Hosted Models
For example:
Llama
Qwen
Mistral
DeepSeek
Deployment:
vLLM
TGI
llama.cpp
Recommended approach:
Application
↓
Pydantic / JSON Schema
↓
Outlines / grammar engine
↓
vLLM / TGI / llama.cpp
↓
LLM
Outlines can construct structural constraints from:
Pydantic
JSON Schema
function signature
for structured generation. Its official documentation also demonstrates generating structure-conforming JSON directly from Pydantic model or JSON Schema.
Therefore typical tech stack:
vLLM
+
Outlines
+
Pydantic
+
application validators
+
observability
You own:
model
inference server
decoding
schema
validator
control over the entire stack.
Advantages:
controllable
customizable
potentially lower large-scale costs
data does not need to be sent to third parties
Cost:
GPU
deployment
scaling
model upgrades
grammar integration
observability
all need to be borne by yourself.
Cloud APIs
For example when using providers like OpenAI / Google, it is recommended to use the provider's native Structured Outputs as much as possible, rather than writing your own:
"Please return JSON"
Architecture:
Application
↓
Pydantic / Zod
↓
JSON Schema
↓
Cloud API
↓
Provider constrained decoding
↓
Structured response
↓
Pydantic Validators
↓
Retry / fallback
For example OpenAI's Structured Outputs can guarantee output matching supported JSON Schema through strict Schema constraints; its official description explicitly distinguishes it from JSON mode.
Google Gemini currently also supports JSON Schema Structured Outputs, and can use Pydantic in Python, Zod in JavaScript to define Schema.
So the cloud approach recommended by the article is essentially:
Native Structured Outputs
+
Pydantic
+
Instructor / retry
+
semantic validators
+
monitoring
rather than:
Prompt
+
regex
+
json.loads()
+
prayer
The conclusion on this is very clear.
Drawing the Complete Production Architecture
Finally, a relatively mature system is roughly:
User Input
│
▼
Prompt / Context
│
▼
JSON Schema
description / enum
required / shallow nesting
│
▼
LLM inference
│
Constrained Decoding
│
▼
Structurally valid JSON
│
▼
Pydantic / Zod
Schema validation
│
┌─────────┴─────────┐
│ │
FAIL PASS
│ │
▼ ▼
Retry loop Business Validator
│
┌──────┴──────┐
│ │
FAIL PASS
│ │
▼ ▼
Retry / Review Semantic Check
│
▼
Downstream
│
▼
Monitoring
And monitoring:
Schema failure rate
Retry rate
Semantic failure rate
Downstream quality
Latency
Token cost
together constitute your reliability loop.
So what this article really wants to change is not just "how to make LLMs output JSON," but an engineering mindset:
Wrong mindset:
LLM
↓
Write Prompt a bit stricter
↓
Hope output is correct
Production mindset:
LLM
↓
Constrained Decoding
↓
Schema
↓
Validation
↓
Retry
↓
Semantic Check
↓
Monitoring
The former depends on model behavior, the latter depends on system mechanisms. The problem of structured output reliability should mainly be solved through architecture, not by constantly changing models or adding "IMPORTANT: STRICTLY RETURN JSON!!!" to Prompts.