Python · Artificial Intelligence · LLM

Pydantic Turns 'Return JSON' From a Prompt Suggestion Into a Schema Check

By databook · Oct 23, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Malformed model output that parses cleanly can sit for months in a database before anyone notices, and neither prompting nor structured-output frameworks can fully prevent it. A schema check at the boundary turns silent corruption into an immediate error, and it keeps working no matter which model provider sits upstream.

Summary

LLM responses to 'return JSON' arrive wrapped in prose, code fences, and unvalidated strings. json.loads() only proves the text is syntactically valid; it does not catch a missing skills key, a phone number stored as '138-0013-8000', or a salary string '15k左右' that Python happily multiplies by 12 and drops into a database column. Failures that throw are easy to find if late; failures that do not throw are worse.

The fix separates extraction from validation. One step strips markdown fences and grabs the outermost curly-brace object; the next runs it through a Pydantic BaseModel whose field declarations carry types, ranges, and required constraints. Custom before-validators normalize uniquely convertible input, so '138-0013-8000' becomes 13800138000 and '15k左右' becomes 15000, while ambiguous input like '五年' fails. Nested models and cross-field validators catch data that is internally inconsistent, such as 9 years of experience when the listed jobs sum to 7.

Retries feed the previous ValidationError back into the model prompt, with different messages for JSON syntax failures vs schema failures, a retry cap, and a fallback return of None. Setting temperature 0 keeps retries stable. LangChain and LlamaIndex offer structured output helpers, but those enforce structure, not business rules, so the final Pydantic check stays as a vendor-neutral boundary. A runnable mock demo requires only pip install pydantic[email].

Takeaways

— json.loads() only validates JSON syntax; it does not check whether fields exist, whether types match, or whether values are within a sensible range.

— Splitting extraction from validation produces two distinct error classes: JSONDecodeError means the model didn't follow the format, ValidationError means a field is wrong; they need different retry messages.

— Pydantic constraints such as Field(gt=0, le=100000), EmailStr, and min_length=1 make missing keys, bad ranges, and empty lists fail at the entry point.

— A mode='before' validator receives raw input before type coercion, so it can normalize unambiguous values: '138-0013-8000' becomes 13800138000 and '15k左右' becomes 15000.

— Ambiguous values such as '五年' should raise a validation error rather than be guessed, because the model's intent cannot be uniquely determined.

— Cross-field validators catch internally invented totals; the example flags 9 years of total experience when the listed jobs sum to 7, with a 0.5-year tolerance.

— A Pydantic validator can only read fields declared earlier in the model, so fields that need cross-checks must be ordered after the fields they depend on.

— Retrying by pasting the validation error into the prompt resolved failures in the demo by the third attempt.

— Retries need a hard cap, a None fallback for manual review or alerts, and temperature 0 so the model doesn't fail differently on each attempt.

— LangChain and LlamaIndex structured-output helpers constrain the JSON shape, not business rules, so the final Pydantic check should remain as the last gate.

Conclusions

The costliest LLM output failure is a valid-looking value that is wrong; Python string multiplication lets '15k左右' pass silently into a salary field, and validation is what makes that class of bug fail loudly at the boundary instead of months later in a report.

Retrying with error messages as context only becomes a reliable loop when the model is deterministic; temperature 0 is the setting that turns the retry strategy from roulette into a convergent process.

Pydantic's field-ordering dependency is an easy trap: cross-field validation only sees earlier fields, so a schema that reads naturally can silently skip a check until the validator runs on incomplete data.

Structured-output frameworks narrow the problem to 'valid JSON' but cannot encode a business's consistency rules; the final schema layer is what survives a model-provider swap.

Concepts & terms

Pydantic

A Python data validation library. A BaseModel class declares fields and constraints with type hints, and Pydantic parses incoming data, raising ValidationError on missing fields, wrong types, or out-of-range values.

JSON Schema

A standard format for declaring the shape of JSON data. Pydantic generates it from a model with model_json_schema(), and placing it in an LLM prompt makes structured extraction more reliable.

function calling

An LLM API capability that returns arguments for a declared function instead of free-form text. It enforces field structure more reliably than a 'return JSON' prompt, but it still does not validate business-level consistency.

mode='before' validator

A Pydantic validator that runs before type coercion, so it sees the raw input. It is useful for normalizing strings like phone numbers, salary ranges, or units into canonical values before the declared type check runs.

From the discussion

The extraction-and-retry pipeline draws a blunt rejection as too clumsy. Its retry logic also contains a concrete error-handling hole: the no-JSON branch raises a bare ValueError, while only ValidationError and JSONDecodeError are caught, so a model's prose-only answer exits the loop instead of triggering another attempt. That gap is fixable by catching the parsing-level ValueError, but the outright dismissal implies the method may not be worth patching.

— The whole extraction-and-retry setup is overengineered and clumsy; the approach is not worth this complexity.

— The retry loop misses the no-JSON case: it catches ValidationError and JSONDecodeError, but the missing-JSON branch throws a plain ValueError, which allows a prose-only model response to break out of the loop.

— The way to cover the 'model only explained itself' failure is to catch the parsing-layer ValueError in addition to the existing exception types.

Featured comments

vgh

Doing it this way is just too dumb.

智测星球

The no-JSON branch throws an ordinary ValueError, but the retry example only catches ValidationError and JSONDecodeError, so in that case it exits the loop immediately. Catching the parsing layer's ValueError as well is necessary to cover the situation where the model only replies with an explanation.

See top comments, translated →

Source: juejin.cn ↗ Google Translate ↗ Backup ↗