When LLMs Hallucinate Code References, Regex and Grep Catch What Self-Review Misses

The Trust Crisis in Multi-Agent Systems

Have you ever encountered this situation: letting one LLM review the output of another LLM?

Suppose you've built a three-role article generation pipeline: Planner is responsible for structuring, Specialist for writing content, and Evaluator for quality review. Sounds perfect, right? But when you scrutinize the Evaluator's work, you'll discover a fundamental contradiction — the Evaluator itself is also an LLM.

It's reviewing an article written by another LLM, like having a suspect judge their own testimony. When the Specialist fabricates a non-existent class name, or references a non-existent file path like /nonexistent/path.py, the Evaluator is very likely to "approve" this content. After all, for an LLM, the boundary between fabrication and fact is inherently blurry.

This is the trust crisis in multi-agent collaboration: we rely on LLMs to verify LLM output, but LLMs themselves hallucinate.

The core value of the crewai-pse project isn't that it's "yet another CrewAI application," but rather that it makes several counter-intuitive design decisions at the architectural level — replacing LLM self-evaluation with programmatic verification, using an independent fix loop to avoid CrewAI overhead, and employing sandbox restrictions to prevent unauthorized access. These decisions reflect deep thinking about the reliability of multi-agent systems.

Design Decision 1: Replacing LLM Self-Evaluation with Programmatic Verification

The Intuitive Approach

Let the Evaluator Agent use grep to check whether code references exist. After all, the Evaluator has a run_bash tool and can execute shell commands.

The Actual Approach

In run.py, the author implemented an independent _verify_article() function that extracts code references using regular expressions and then verifies them with filesystem grep:

def _verify_article(article: str, source_dir: Path) -> tuple[list[str], list[str]]:
    """Programmatic verification: grep checks whether code references exist. Returns (fictitious list, verified list)."""
    refs = set(re.findall(r"`([A-Za-z_][\w._]*(?:/[A-Za-z_][\w._]*)*)`", article))
    for match in re.finditer(r"```(?:python)?\s*\n(.*?)```", article, re.DOTALL):
        for line in match.group(1).split("\n"):
            m = re.match(r"^\s*(?:def|class)\s+(\w+)", line)
            if m:
                refs.add(m.group(1))

This function does two key things:

Extracts all backtick-wrapped code identifiers: including file paths, function names, class names
Extracts def and class declarations from code blocks: ensuring key identifiers in code examples are also verified

Then, it verifies each reference:

    for ref in sorted(refs):
        if len(ref) < 3 or ref.startswith("http"):
            continue
        if ref in PYTHON_KEYWORDS:
            continue
        # File path → recursively search disk (search .py and .md files)
        if "/" in ref or ref.endswith(".py") or ref.endswith(".md"):
            found = any(
                f.name == ref.rsplit("/", 1)[-1]
                for ext in ("*.py", "*.md")
                for f in source_dir.rglob(ext)
                if ".venv" not in str(f) and "__pycache__" not in str(f)
            )

Why It's Better

There are several subtle design details here:

1. Deterministic Checking vs. Probabilistic Judgment

Regex matching and file search are deterministic. If _verify_article() says a certain class name doesn't exist, then it truly doesn't exist. This stands in stark contrast to the LLM's "probabilistic judgment" — the LLM might "feel" that a class name exists, but in reality it has only seen similar naming patterns.

2. Distinguishing "Code References" from "Python Keywords"

    PYTHON_KEYWORDS = {
        "def", "class", "import", "from", "return", "yield", "raise",
        "try", "except", "finally", "with", "as", "if", "elif", "else",
        "for", "while", "break", "continue", "pass", "lambda", "and",
        "or", "not", "in", "is", "True", "False", "None", "self", "cls",
        # ... more keywords and built-in functions
    }

Without excluding these keywords, words like def or class appearing in the article would be falsely flagged as fictitious code references.

3. Recursive Search Excluding Virtual Environments

if ".venv" not in str(f) and "__pycache__" not in str(f)

This is a very practical detail. Virtual environments and cache directories should not be included in the verification scope, otherwise they would generate a large number of false positives.

Design Decision 2: Independent Fix Loop, Bypassing CrewAI Orchestration

The Intuitive Approach

When the Evaluator discovers a problem, the Planner re-plans, and the Specialist rewrites. Let CrewAI's orchestration engine handle the entire fix flow.

The Actual Approach

In the main loop of run.py, the author uses an independent OpenAI client to directly call the LLM for fixes, completely bypassing Crew orchestration:

    # Programmatic verification + automatic correction
    max_retries = 3
    for attempt in range(1, max_retries + 1):
        fictitious, verified = _verify_article(article, source_dir)
        print(f"\n{'='*60}")
        print(f"  Verification (Attempt {attempt}) — {len(verified)} items verified")
        if fictitious:
            # Separate code references and exaggerated terms
            code_refs = [f for f in fictitious if not f.startswith("[Exaggerated]")]
            exaggerations = [f for f in fictitious if f.startswith("[Exaggerated]")]

            print(f"  ❌ Fictitious content {len(fictitious)} items: {', '.join(fictitious)}")
            if attempt < max_retries:
                print("  🔄 Auto-correcting...")
                fix_parts = []
                if code_refs:
                    fix_parts.append(f"**Fictitious code references (do not exist in source code, must be deleted)**: {', '.join(code_refs)}")
                if exaggerations:
                    exagg_words = [f.split("—")[0].replace("[Exaggerated]", "").strip() for f in exaggerations]
                    fix_parts.append(f"**Prohibited exaggerated terms (must be completely removed from the article)**: {', '.join(exagg_words)}")

                fix_prompt = f"""The following article has been found to have issues during verification. Please correct it.

{'\n'.join(fix_parts)}

**Rules**:
1. Fictitious code references: Delete the sentences or code examples containing the reference, do not creatively replace
2. Exaggerated terms: Delete the entire sentence containing the term, do not attempt to rewrite
3. Keep the rest of the article unchanged
4. Output the corrected complete article (starting from Front Matter), do not output explanations

## Current Article
{article}"""
                try:
                    resp = fix_client.chat.completions.create(
                        model=fix_model,
                        messages=[{"role": "user", "content": fix_prompt}],
                        max_tokens=8192,
                        temperature=0.7,
                    )
                    article = resp.choices[0].message.content

Why It's Better

1. Performance Advantage

Re-running the complete three-agent pipeline (Planner → Specialist → Evaluator) for every fix is very expensive. Directly calling the LLM API for fixes requires only one API call instead of three.

2. Precise Control

The fix prompt can directly specify "delete fictitious content" rather than "creatively replace." This prevents the LLM from introducing new hallucinations during the fix process.

3. Programmatic Fallback

    EXAGGERATED_TERMS = {
        # Exaggerated terms have been removed as required
    }

def _strip_exaggerated(text: str) -> str:
    """Programmatically delete sentences containing exaggerated terms (split by period/newline)."""
    for keyword in EXAGGERATED_TERMS:
        # Clean sentence by sentence based on sentence boundaries (Chinese period, newline, semicolon)
        lines = text.split("\n")
        cleaned = []
        for line in lines:
            if keyword in line:
                # Try to delete only the clause containing the keyword (split by Chinese punctuation)
                parts = re.split(r'([。；;])', line)
                filtered = []
                for i in range(0, len(parts) - 1, 2):
                    sentence = parts[i]
                    punct = parts[i + 1] if i + 1 < len(parts) else ""
                    if keyword not in sentence:
                        filtered.append(sentence + punct)
                # Handle the last segment (no punctuation ending)
                if len(parts) % 2 == 1 and parts[-1]:
                    if keyword not in parts[-1]:
                        filtered.append(parts[-1])
                cleaned_line = "".join(filtered).strip()
                if cleaned_line:
                    cleaned.append(cleaned_line)
            else:
                cleaned.append(line)
        text = "\n".join(cleaned)
    return text

When LLM fixes also fail (up to 3 rounds), the programmatic fallback function _strip_exaggerated() directly deletes sentences containing exaggerated terms. This is the last line of defense, ensuring the output does not contain unverified feature descriptions.

Design Decision 3: Sandboxed File Access

The Intuitive Approach

Give Agents unrestricted filesystem permissions, allowing them to freely read any file.

The Actual Approach

In tools.py, the read_file tool forces the path to be under PSE_ROOT:

_PROJECT_ROOT = Path(os.getenv("PSE_ROOT", Path.cwd())).resolve()

@tool("read_file")
def read_file(path: str) -> str:
    """Read file content. Parameter path is the file path (restricted to project directory)."""
    p = Path(path).resolve()
    if not str(p).startswith(str(_PROJECT_ROOT)):
        return f"[Error] Path outside project scope: {path}"
    if not p.exists():
        return f"[Error] File does not exist: {path}"
    if not p.is_file():
        return f"[Error] Not a file: {path}"
    return p.read_text(encoding="utf-8")

Why It's Better

1. Security

Prevents the Agent from reading sensitive files outside the project. Even if the Agent is induced to read /etc/passwd or the user's private key, read_file will refuse access.

2. Controllability

The root directory is flexibly configured via the PSE_ROOT environment variable. This allows the same tool to be reused across different projects.

3. Handling Symbolic Links

_PROJECT_ROOT = Path(os.getenv("PSE_ROOT", Path.cwd())).resolve()

Using Path.resolve() handles symbolic links, preventing path escape. If _PROJECT_ROOT is a symbolic link, resolve() will resolve to the actual path, ensuring the security check is effective.

Complete Pipeline: From Source Code to Bilingual Article

Now let's see how the entire pipeline operates:

Source Code → Planner Outline → Specialist Draft → Verification/Fix → Chinese Final → English Translation

CrewAI Phase

In agents.py, three Agents are created and assembled into a Crew:

def create_crew(task: str | None = None) -> Crew:
    """Create PSE three-role Crew (Sequential process)."""
    return Crew(
        agents=[create_planner(task), create_specialist(task), create_evaluator(task)],
        process=Process.sequential,
        verbose=True,
    )

Note the use of Process.sequential, meaning Agents execute in order: Planner plans first, then Specialist writes. Although the Evaluator is created, it is not actually used in the main flow — verification work is handled by the programmatic _verify_article() function.

Programmatic Verification Phase

After the Specialist outputs the draft, the main loop in run.py begins verification:

    crew_output = crew.kickoff()

    # Extract Specialist's output from CrewOutput
    if crew_output.tasks_output:
        article = crew_output.tasks_output[-1].raw
    elif specialist_task.output:
        article = specialist_task.output.raw
    else:
        article = ""

Then it enters the verification loop until all fictitious content is cleared or the maximum retry count is reached.

Translation Phase

After the Chinese final draft, it is automatically translated into English:

    translate_prompt = (
        "Translate the following Chinese technical article to English. "
        "Keep ALL code examples, file paths, class names, and function names unchanged. "
        f"Output ONLY the translated article:\n\n{article}"
    )

The translation prompt specifically emphasizes "Keep ALL code examples, file paths, class names, and function names unchanged," ensuring technical accuracy is not compromised during translation.

Design Philosophy Summary

The core viewpoint of crewai-pse is clear: the reliability of multi-agent systems comes not from more Agents, but from smarter verification mechanisms.

This project demonstrates that when LLM output requires factual accuracy guarantees, programmatic verification is more reliable than LLM self-evaluation. Regular expressions and file search are deterministic and will not deviate based on the LLM's "confidence level."

Applicable Scenarios

Any AI application requiring a "generation + verification" closed loop can draw on these design decisions:

Code Generation: Verify whether generated code references APIs that actually exist
Technical Documentation: Ensure examples and references in documentation are consistent with the actual codebase
Research Reports: Prevent models from fabricating non-existent experimental results or references

Insights

Don't over-rely on LLM self-evaluation: Having an LLM check another LLM's output is unreliable
Programmatic verification is key: Replace probabilistic judgment with deterministic checks
Fix mechanisms independent of orchestration: Bypass complex Agent flows and use LLM APIs directly for precise fixes
Security sandboxes are essential: Even for "assistant" type Agents, restrict their filesystem access permissions

crewai-pse is not meant to replace CrewAI or other multi-agent frameworks, but rather demonstrates how to add a layer of reliable verification mechanisms on top of these frameworks. This is a pragmatic approach: acknowledge the limitations of LLMs, compensate for them with engineering means, rather than expecting the models themselves to become "perfect."

Source Code Navigation

Module	File	Description
Main Pipeline	run.py	Core orchestration: CrewAI invocation, programmatic verification, fix loop, translation
Verification Logic	run.py::_verify_article	Regex extraction of references + filesystem grep verification
Agent Definitions	agents.py	Creation of Planner/Specialist/Evaluator and Crew assembly
Sandbox Tools	tools.py	Implementation of read_file's sandbox path restriction
Planner Prompt	planner.md	Instruction templates for five narrative styles
Specialist Prompt	specialist.md	Source code verification requirements and writing specifications
Evaluator Prompt	evaluator.md	Verification steps and judgment format
Configuration Management	config.py	Environment variable loading

Quick Start

Installation

pip install crewai crewai-tools python-dotenv openai

Configuration

Copy .env.example to .env, fill in your OpenAI API Key
Set the PSE_ROOT environment variable to point to your project root directory
Add project configuration in tasks/project-articles/projects.json

Running

cd tasks/project-articles
python run.py <project-name>

For example:

python run.py my-project

This will automatically generate a Chinese technical article for that project, along with an English translation.

This article was generated by the crewai-pse framework itself — the Planner structured it, the Specialist wrote the content, and programmatic verification ensured the accuracy of all code references and class names.

Series Articles

Frequently Asked Questions (FAQ)

Q: Why use programmatic verification instead of LLM self-evaluation? LLM self-evaluation has confirmation bias and tends to give itself a "pass"; programmatic verification uses deterministic assertions (type, structure, rule matching) to judge whether output is qualified, with reproducible, debuggable results that better expose real errors.

Q: Why an independent fix loop, bypassing CrewAI's native orchestration? CrewAI's built-in loops are difficult to precisely control the rhythm and count of "verification failure → fix → re-verification." Making the fix loop independent allows the framework to explicitly trigger Fix when verification fails, rather than relying on the orchestrator's vague retries.

Q: How is file access sandboxed? Agents can only access pre-declared directories and operation types; unauthorized reads/writes are rejected. This preserves the Agent's ability to handle files while limiting the damage scope to within the sandbox.

Q: What can this pipeline ultimately produce? Automatically generate bilingual (Chinese + English) technical articles from source code: the CrewAI phase handles analysis and drafting, the programmatic verification phase acts as a quality gate, and the translation phase produces the English version, all without manual sentence-by-sentence proofreading.

Q: What is the core of PSE's design philosophy? Verifiability over fluency. Rather than pursuing LLM output that "looks right," use engineering constraints (programmatic verification, sandboxing, explicit state) to ensure it "is right," building trust on observable mechanisms.

When LLMs Hallucinate Code References, Regex and Grep Catch What Self-Review Misses

The Trust Crisis in Multi-Agent Systems

Design Decision 1: Replacing LLM Self-Evaluation with Programmatic Verification

The Intuitive Approach

The Actual Approach

Why It's Better

Design Decision 2: Independent Fix Loop, Bypassing CrewAI Orchestration

The Intuitive Approach

The Actual Approach

Why It's Better

Design Decision 3: Sandboxed File Access

The Intuitive Approach

The Actual Approach

Why It's Better

Complete Pipeline: From Source Code to Bilingual Article

CrewAI Phase

Programmatic Verification Phase

Translation Phase

Design Philosophy Summary

Applicable Scenarios

Insights

Source Code Navigation

Quick Start

Installation

Configuration

Running

Series Articles

Frequently Asked Questions (FAQ)

Related Reading