1. Where the problems hide when you don't validate

Processing the same model response with the most naive approach runs into four situations in a row:

json.loads(llm_text)              # ① wrapped in extra prose
data['years_experience'] + 1      # ② value is '五年'
len(data['skills'])               # ③ key does not exist at all
'15k左右' * 12                    # ④ compute annual salary

Actual output:

JSONDecodeError: Expecting value: line 2 column 1 (char 1)
TypeError: can only concatenate str (not "int") to str
KeyError: 'skills'
'15k左右' * 12 = 15k左右15k左右15k左右…(repeated 12 times)

Cases ①②③ are still lucky: the program blows up in front of you, at worst a little far from the source. A KeyError often surfaces in a downstream line that generates a report, several functions away from the parsing code.

The truly troublesome case is ④. String multiplication is a perfectly legal operation in Python; it does not raise an exception and it does not write a log. The result is a string no one has seen before, and it goes straight into your annual-salary field.

What these cases have in common: the problem is either discovered too late, or never discovered at all.

json.loads only guarantees 'this is valid JSON syntax.' It says nothing about whether fields exist, whether types are right, or whether values are plausible.

2. Adding validation is really adding two gates

Once validation is added, the pipeline changes from 'a text blob straight into business logic' to two gates: first pull the JSON out of the surrounding prose, then validate it field by field against the schema.

The first gate is simple: strip the markdown fences, then use a regex to grab the outermost pair of curly braces.

def extract_json_from_llm_response(response: str) -> dict:
    text = re.sub(r'```(?:json)?', '', response)      # strip ``` fences
    match = re.search(r'\{.*\}', text, re.DOTALL)     # grab outermost braces
    if not match:
        raise ValueError('No JSON found in the response')
    return json.loads(match.group())

The second gate goes to Pydantic.

Keeping the two layers separate is valuable because of what the errors tell you. A JSONDecodeError means 'the model did not follow the format.' A ValidationError means 'the format was right but the content is wrong.' These two situations need completely different handling (section 5 uses that distinction), so don't stuff them into the same except Exception.

3. What the contract looks like: types, ranges, formats

Define a Resume with fields and constraints written together:

class Resume(BaseModel):
    name: str
    email: EmailStr
    phone: str
    city: str
    expected_salary: int = Field(..., gt=0, le=100_000)
    skills: List[str] = Field(..., min_length=1)
    experiences: List[WorkExperience] = Field(..., min_length=1)
    years_experience: float = Field(..., ge=0, le=40)

EmailStr validates the email format directly (you need email-validator, the extra that comes with pydantic[email]);

Field(gt=0, le=100_000) puts the allowed value range into the type annotation;

Fields are required by default; missing any one triggers a Field required error immediately, and min_length=1 guarantees lists are not empty.

These few lines already stop case ③ from section 1: a missing field fails at the entrance instead of throwing a KeyError downstream.

The interesting part is custom validators. If a value has an unambiguous deterministic conversion, don't reject it; normalize it on the way in:

@field_validator('phone')
@classmethod
def normalize_phone(cls, v: str) -> str:
    digits = re.sub(r'\D', '', v)                      # remove '-' and spaces
    if digits.startswith('86') and len(digits) == 13:  # remove +86 prefix
        digits = digits[2:]
    if len(digits) != 11 or not digits.startswith('1'):
        raise ValueError(f'Not a valid mobile number: {v!r}')
    return digits

@field_validator('expected_salary', mode='before')
@classmethod
def parse_salary(cls, v):
    # Convert '15k' / '1.5万' to a number first, then hand it to int validation.
    ...

mode='before' means 'run me before type validation,' so it gets to see the raw string.

Running it gives:

Phone '138-0013-8000' -> 13800138000
Salary '15k左右'       -> 15000

Input that is messy in format but clear in meaning is rescued; the database always stores a normalized 11-digit phone number and an integer amount.

Only what cannot be rescued raises an error. For example, '五年': you cannot and should not guess on the model's behalf whether it means 5 or 5.5, so let it fail:

Input should be a valid number, unable to parse string as a number [input_value='五年']

This message contains the field name, the reason, and the original value, which makes it exactly what you can feed back to the model (section 5).

There is one decision rule: is the conversion rule uniquely determined? If yes, convert to the canonical value; if not, raise an error.

4. Nesting and cross-field checks: catch invented numbers

Real data is rarely flat.

A resume contains a list of work experiences, so define a sub-model and write List[WorkExperience] in the main model:

Pydantic recursively constructs sub-objects, and when a field in one entry fails, the error message points precisely to, say, 'the start of experience entry 2'.

But range and format validation only stops format errors; it does not stop the model from inventing data with a straight face. For example, three listed jobs add up to 7 years, but it writes 9 as the total years.

That is when you cross-check fields:

@field_validator('years_experience')
@classmethod
def check_years_match(cls, v: float, info) -> float:
    experiences = info.data.get('experiences') or []
    if experiences:
        total = sum(_years_of(e.start, e.end) for e in experiences)
        if abs(total - v) > 0.5:                       # allow 0.5 years of tolerance
            raise ValueError(f'Total work years {v} does not match the sum of experience entries {total:.2f}')
    return v

The output is:

Value error, total work years 9.0 does not match the sum of experience entries 7.00 years

This kind of 'internally inconsistent with itself' problem is the category that needs the closest attention in LLM output. Every field looks legal on its own, but the combination is false.

The same idea applies in many places: the sum of order line amounts must equal the order total, the average order value in a daily report must equal revenue divided by order count, and an extracted date range must not be inverted.

There is a trap when writing this: a validator can only read fields that were already validated before it. So years_experience must be declared after experiences; otherwise info.data is empty.

Also, don't set the tolerance to 0; the model's calculated total is often off by a few tenths.

5. After failure: feed the error back to the model as-is

What do you do after validation fails? The simplest and most effective approach is to put the previous error message into the prompt and let the model answer again.

def extract_with_retry(llm_call_function, max_retries: int = 3):
    last_error = None
    for attempt in range(max_retries):
        try:
            return Resume(**extract_json_from_llm_response(llm_call_function(last_error)))
        except ValidationError as e:
            last_error = str(e)                        # schema error: include specific field problems
        except json.JSONDecodeError:
            last_error = 'The response is not valid JSON. Output only JSON, no explanatory text.'
        if attempt == max_retries - 1:
            return None                                # retries exhausted, hand off to upstream fallback

llm_call_function(last_error) is responsible for splicing the error into the prompt, so the model knows where to fix things.

Real output from three rounds on the same resume text:

Attempt 1 failure: Input should be a valid number ... ('五年')
Attempt 2 failure: Total work years 9.0 does not match the sum of experience entries 7.00 years
Success: Chen Shuya, 7.0 years experience, expected 15000 yuan, skills ['Python', 'SQL', 'FineBI']

Three details to note:

1. Use different prompts for JSON errors and schema errors. The former means 'you did not follow the format'; the latter means 'a particular field is wrong.'

2. Always set a limit. When the retries run out, return None and let the upper layer degrade gracefully (such as a manual review queue or log alerts) instead of crashing the whole pipeline.

3. Set temperature to 0 when calling; otherwise the same prompt fails at a different spot each time and retrying is wasted effort.

6. Connecting to a real model call

The previous examples used simulated responses. Switching to a real API only adds two steps: put the schema into the prompt, and hand the return value to the same parse_resume.

schema_hint = json.dumps(Resume.model_json_schema(), ensure_ascii=False)
response = client.chat.completions.create(
    model='gpt-4o-mini', temperature=0,
    messages=[{'role': 'user', 'content': f'Extract according to this JSON Schema:\n{schema_hint}\n\n{text}'}])
return parse_resume(response.choices[0].message.content)

Using a framework is even easier: LangChain has llm.with_structured_output(Resume) (uses native function calling, recommended) and PydanticOutputParser (prompt-guided, works with any model); LlamaIndex has LLMTextCompletionProgram.from_defaults(output_cls=Resume).

But what frameworks help constrain is the structure, not the business semantics. function calling can guarantee the fields and types are right, but it cannot guarantee business-specific rules like 'total years equals the sum of experience entries.'

Also, structured output can still be non-compliant in edge cases.

So keep that final Pydantic validation layer. It also serves as a unified exit when you connect to any model vendor: changing models does not require changing downstream code.

7. Complete code

The complete runnable script is shared at: resume_extraction_demo.py https://url11.ctfile.com/f/45455611-17569905548293-8f906d?p=6872

Every 'model response' inside it is a simulated string. Install pydantic[email], run python resume_extraction_demo.py, and you will see every set of output from this text.

The simulated code execution result is shown below:

To adapt it to your own scenario:

Step 1, change the model. Replace the fields of Resume with yours: document date, amount, category, ID, owner. First distinguish 'must have' from 'may have.' For the latter, write Optional[str] = None; don't make everything required, or the model will get rejected every time. Add gt / ge / le to every numeric field; these constraints are the cheapest and most effective.

Step 2, change the parsing. Copy extract_json_from_llm_response and the layered exceptions directly. Put the output of model_json_schema() in the prompt and set temperature=0.

Step 3, change the fallback. Failure -> retry 2 to 3 times -> if it still fails, return None and route to manual handling or degradation. Don't let a single parsing failure take down the whole batch job.

Back to that 'return JSON' promise at the opening. Whether the model makes mistakes is out of your control; whether the mistake is stopped on the spot or quietly flows into the database is entirely up to you.

The difference is whether the contract is written down as code. Once it is, it becomes a check that executes on every call, not just a verbal agreement that only takes effect inside the prompt.