When LLMs Hallucinate Code References, Regex and Grep Catch What Self-Review Misses
LLM-as-judge patterns are spreading fast in agent frameworks, but they inherit the same hallucination problem they're supposed to catch. Swapping in deterministic verification — regex plus a filesystem grep — costs almost nothing and eliminates an entire class of silent failures where fabricated API names or file paths sail through review.
Multi-agent pipelines that let one LLM review another's output suffer a fundamental trust problem: the reviewer hallucinates just as easily as the writer. crewai-pse sidesteps this by pulling verification out of the LLM entirely. A `_verify_article()` function extracts every backtick-wrapped identifier and `def`/`class` declaration from the generated text, then greps the actual source tree to confirm existence — a deterministic check that cannot be fooled by an LLM's false confidence. Python keywords and virtual-environment paths are explicitly excluded to avoid false positives.
When fictitious references are found, the pipeline does not re-run the full Planner–Specialist–Evaluator crew. Instead, it calls the OpenAI API directly with a fix prompt that instructs deletion, not creative replacement, keeping the repair cheap and controlled. If three LLM fix rounds still leave exaggerated claims, a programmatic fallback strips sentences containing banned terms by splitting on Chinese punctuation boundaries.
File access is sandboxed: the `read_file` tool resolves paths and rejects any request outside `PSE_ROOT`, blocking symlink escapes and access to `/etc/passwd` or private keys. The full pipeline runs source code through CrewAI for drafting, then through verification, fix loops, and finally translation into English with explicit instructions to preserve all code identifiers unchanged.
The Evaluator agent is created and assembled into the Crew but never actually used for verification — its role is purely ceremonial, while the real quality gate is the programmatic function. This is a quiet admission that LLM self-review adds so little trust that it's better to just not call it.
Bypassing CrewAI's orchestration for the fix loop isn't just about saving API calls. It also avoids the framework's fuzzy retry semantics, giving the developer exact control over when and how a fix is triggered.
The fix prompt's insistence on deletion over creative replacement targets a specific failure mode: LLMs asked to 'correct' hallucinations often invent new plausible-sounding but equally false substitutes.
Sandboxing file access at the tool level rather than relying on agent prompts or system messages is a defense-in-depth move — no matter what the LLM is tricked into requesting, the tool itself refuses.
I feel like adding to the agent's soul md: must not fabricate, must not take things for granted, must not assume — it feels like that already improves things a lot.
Indeed effective 👍 My article actually takes it one step further: MD is responsible for constraints, Programmatic Verification is responsible for verification. If a program can judge it, don't let the Agent say "I didn't fabricate" by itself.