跪拜 Guibai
← Back to the summary

The Engineering Boundaries That Make AI Agents Controllable

It's been a while since we talked about Agents. Today, let's continue discussing the key implementation points of Agents. In fact, the main issues with an Agent in daily production are controllability and reliability. So, although theoretically an Agent is just a simple combination of LLM + Tools + RAG, in reality, many fundamental details need to be handled across different scenarios, such as:

We've actually discussed these issues many times. Basically, a complete Agent includes at least six types of capabilities:

Goal Understanding, Task Planning, Tool Use, State & Memory, Execution Feedback, Security & Evaluation.

The AI model is indeed the key decision-making component within an Agent, but what determines whether the system is controllable and reliable is almost entirely the external engineering boundaries.

For example, suppose you want to build a "Contract Assistant" AI Agent. A user might then propose:

Find the contract the boss sent last week, check the payment terms, summarize them into a paragraph, and prepare it to be sent to finance.

This scenario requires an entire task chain, such as:

So, how do you think this task should be handled appropriately? Generally, the Agent flow for a "Contract Assistant" should look something like this, combining a dynamically planning Agent with fixed-process workflows:

This is actually the most common hybrid implementation in a production environment:

The Workflow determines the executable track, and the Agent is responsible for semantic judgment and dynamic selection within that track.

Then, looking at the previous diagram from both an external and internal perspective, it can be broken down into multiple layers:

Basically, even the most basic Agent will possess these layered capabilities. Because to be reliable and controllable in production, the Agent Loop cannot just be a simple "Think-Act-Observe" loop. In business, it definitely needs:

Understand Goal
  → Read State and Context
  → Generate Candidate Plans
  → Select Constrained Tools
  → Parameter and Permission Validation
  → Request Confirmation if Necessary
  → Execute Tool
  → Normalize Results
  → Verify if Completion Conditions are Met
  → Re-plan if Not Complete, Output if Complete

Then, in this chain, failure can occur at every step, so you need to treat errors as normal branches, ensuring at least a result feedback every time.

1. Intent Recognition

This is a perennial topic. A user's question, like the earlier "Find the contract the boss sent last week, check the payment terms, summarize them into a paragraph, and prepare it to be sent to finance," definitely cannot be sent directly to the model for processing.

The Agent must first perform intent recognition. The goal of intent recognition is to make trade-offs and classifications between accuracy, latency, cost, and risk. Here is a common intent recognition classification scenario:

That is, you need to decide how to categorize based on your business needs. An Agent like the "Contract Assistant" clearly has deterministic rules:

The rule layer must be explicit and streamlined. Its main purpose is to quickly intercept the most definite batch of requests. If there are too many rules, it will turn into a mess and might evolve into an unmaintainable collection of keywords and if-else statements.

So this layer needs to route based on scenarios like the previous conversation turn, the current page and selected object, and the current task node. This means using some lightweight local small text models combined with historical information to quickly make semantic intent judgments.

    {
      "intent": "review_contract_and_prepare_email",
      "entities": {
        "sender_role": "manager",
        "time_range": "last_week",
        "document_type": "contract",
        "target_audience": "finance"
      },
      "missing_fields": [],
      "risk_level": "medium",
      "confidence": 0.87,
      "next_action": "search_mail"
    }

In intent recognition, there's always a confidence score. Its role is to decide the next step. Generally, the confidence score can trigger at least three different strategies:

For intents like "cancel order" and "cancel appointment," we certainly can't rely on the model to gamble on a 90% success rate. High-risk operations definitely require a human-in-the-loop. Even if matched, they need to enter a clarification stage.

2. Orchestration and State Machine

After the Agent knows the user's intent, it must also know which step it's on. You can't just use the historical conversation to judge this state, because chat logs only have input history and lack a complete task state. Therefore, a reliable Agent State needs to contain at least something like:

    from typing import Any, Literal, TypedDict
    
    class AgentState(TypedDict):
        task_id: str
        user_goal: str
        current_step: str
        status: Literal["running", "waiting_user", "completed", "failed"]
        collected_slots: dict[str, Any]
        selected_tools: list[str]
        observations: list[dict[str, Any]]
        evidence: list[dict[str, Any]]
        retry_count: dict[str, int]
        token_budget: int
        time_budget_ms: int
        pending_approval: dict[str, Any] | None

Here, we need to save both raw data and structured results in the state as much as possible to achieve controllability. A markdown output definitely won't work; structured data is needed so that the same facts and state can be reused across different nodes.

For instance, the biggest role of the state is to clarify and define completion conditions. Every task must have explicit completion conditions:

For example, the completion conditions for the "Contract Assistant" must include: the contract source has been confirmed, whether the payment terms have original text evidence, the summary and evidence must be consistent, the email draft has been generated, and whether the sending has been executed.

Then, it also needs to support 'checkpoint and persistence' capabilities, because long task processes will inevitably have pauses and resumptions, such as:

Supporting the Agent in saving these checkpoints is also a critical requirement. Generally, at least the following needs to be saved: state version, completed nodes, pending execution nodes, tool result summaries, idempotency keys, and approval information.

Finally, there is the support for 'recovery and execution'. The most important thing here is to 'prevent duplicate execution.' Unlike a Coding Agent, where recompiling and re-scanning is not a big deal, this is definitely not acceptable for something like a "Contract Assistant." When the system recovers from a checkpoint, a node might have already run, but we hadn't waited for the result. Therefore, all operations with side effects must consider idempotency:

This part is actually not directly related to the Agent; it's more about the business backend's capabilities and state synchronization design.

3. Tool System

Tools are essentially the business abstraction within an Agent and represent the Agent's value and intellectual performance. However, defining a tool isn't just about writing a function call; it also requires a reliable process to support it:

Generally, a tool in an Agent is like a contract. You need to define and prepare a complete set of definitions for the tool, such as:

For example, a tool definition could look like:

    from typing import Literal
    from pydantic import BaseModel, Field
    
    class SearchMailArgs(BaseModel):
        sender_role: Literal["manager", "finance", "customer", "unknown"]
        start_date: str = Field(description="ISO-8601 date")
        end_date: str = Field(description="ISO-8601 date")
        attachment_type: Literal["contract", "invoice", "any"] = "contract"
        max_results: int = Field(default=20, ge=1, le=50)
    
    class ToolProposal(BaseModel):
        tool_name: Literal["search_mail", "read_attachment", "extract_clause", "create_draft"]
        arguments: dict
        expected_result: str
        risk_level: Literal["read", "write", "high_risk"]

Of course, here the Schema only guarantees the structure of the execution result, but whether the result is correct still requires subsequent judgment. For example: a date might be in a legal format but out of range, an amount might be a number but exceed permissions, an email might be valid but belong to the wrong recipient. These all require business validation after the Schema check.

Next is tool matching and retrieval. The intent decomposition we discussed earlier is largely aimed at matching the right tools. Therefore, tools should be as precise as possible. If the model's capability isn't top-tier and rules can't achieve fine-grained tool distinction, then the number of tools must not be too large.

For instance, exposing dozens or hundreds of tools to the model at once mostly just increases the chance of misselection and parameter confusion. So, generally, an Agent will first retrieve tools and then register them for the large model in this task. For example, you can:

Then, before execution, format validation, business validation, permission validation, and risk confirmation are all required. This is also the basic etiquette of tool execution.

    def execute_tool(proposal, context):
        args = schema_registry.validate(proposal.tool_name, proposal.arguments)
        policy.authorize(context.user, proposal.tool_name, args)
        policy.validate_business_rules(context, proposal.tool_name, args)
    
        if policy.requires_confirmation(proposal.tool_name, args):
            return pause_for_human_approval(proposal, args)
    
        idempotency_key = build_idempotency_key(
            context.task_id, proposal.tool_name, args
        )
    
        return executor.run(
            tool_name=proposal.tool_name,
            args=args,
            idempotency_key=idempotency_key,
            timeout_ms=10_000,
        )

Additionally, just like the intent confidence and risk mentioned earlier, tools also need a risk rating. Or rather, the intent risk mentioned before basically comes from the tool risk. For example:

Level Typical Operations Default Strategy
R0 Pure Reasoning Classification, summarization, rewriting Can execute automatically
R1 Read-only Query orders, search emails, read documents Auto-execute after authorization, log audit
R2 Reversible Write Create drafts, add tags, save temporary configs Can auto or light confirm, must be idempotent
R3 High-risk Write Send emails, refund, delete, transfer, execute scripts, modify production configs Strong confirmation, least privilege, full audit
R4 Forbidden or Isolated Exceeding business authorization, cross-tenant access, sensitive batch operations Directly reject or transfer to human

Even on the user UI, we might need to display things like: what tool will be called next, which object it will operate on, the scope of impact, whether it's reversible, and what the key parameters are.

Then, tools will inevitably fail during execution. The failure errors here must also be structured, for example:

    {
      "status": "error",
      "error_type": "MISSING_ARGUMENT",
      "retryable": false,
      "field": "contract_id",
      "message": "Missing contract identifier, requires user to select a candidate contract",
      "suggested_action": "ask_user"
    }

Common error classifications are generally:

With error classification and rating, you can then retry errors. The number of retries can also be configured based on the tool's idempotency, error type, latency budget, and business consequences.

High-risk operations definitely allow zero retries.

4. Structured Output

We've been emphasizing that structured data needs to flow and be held internally within the Agent. However, getting structured output from a model also has its nuances. Common problems when an LLM outputs JSON include:

Interleaved explanatory text, Markdown code blocks, missing fields, type errors, truncation, out-of-bounds enums, and semantic errors.

These are all very common problems. Therefore, compatibility and automatic error handling require a reliable JSON flow to support, such as:

Explicit Output Contract
  → Model-native Structured Output / Function Calling
  → Check stop reason and truncation
  → JSON Parsing
  → Schema Validation
  → Business Semantic Validation
  → Limited Repair
  → Retry with Error Information
  → Degradation or Task Splitting

Generally, it's definitely preferable to use the model's native structural constraints first. For example, for models that support strict JSON Schema, Function Calling, or Structured Output, you certainly cannot rely on prompting with "only output JSON," which is a completely unreliable promise.

But generally, we must still distinguish three states of correctness:

Strict Schema validation mainly solves the first two; semantic correctness requires subsequent business review.

Then, for syntax and structural correctness issues, errors will definitely occur during Agent operation. So, we can add multiple methods for repair and retry, but don't guarantee a fix can always be made, otherwise it's easy to get into an infinite loop.

In fact, for JSON structural problems, the ones that can be safely repaired are usually only clear formatting issues, such as:

But if you encounter these problems, repair is generally not recommended, for example:

In these cases, you should directly judge it as a failure and then re-request or split the task for a retry, directly ignoring this JSON result.

Additionally, it's not recommended to use an Agent to process long and deeply nested structured data. If the data hierarchy is long and deep, it needs to be split. For example, you can break it down into:

Otherwise, the probability of truncation and problems will be very high. After splitting, each sub-step can also be independently validated and evaluated.

5. RAG

RAG is also a perennial topic. According to common rules for Agents, it can generally be divided into:

That is to say, although we might simply say RAG is just "Vector DB + Top K + LLM," in an actual production chain, it generally includes at least:

Parsing, Chunking, Indexing, Query Understanding, Multi-path Recall, Deduplication, Reranking, Context Assembly, Generation, and Citation.

For example, document parsing: a contract is a very typical scenario. It needs to preserve the original structure, because while ordinary text can be quickly chunked by headings, paragraphs, and length, contracts, technical specifications, policy documents, tables, and scanned copies require more fine-grained structured parsing.

So for documents, a Chunk needs to save more than just the body text:

Then, layer resources based on business needs, for example, "70% ordinary documents, 30% high-value documents." This needs to be experimented with based on your actual business situation.

Then, the next two things to do are: Recall and Reranking.

In the recall phase, you can generally do addition, mainly to avoid missing key evidence. For example, you can combine:

Then, the reranking phase needs to do subtraction. After recalling many candidates, we need to use a Reranker and rule-based features for sorting, such as:

Finally, there is context assembly. Once everything is retrieved, you can assemble the context state. Here, some secondary optimizations can generally be done, such as:

For example, for a contract assistant, the retrieval results must be traceable back to a specific contract, page, and clause. You cannot just give the model a piece of text that has lost its source.

6. Memory

Memory in an Agent can generally be divided into short-term state, long-term facts, and external knowledge. Structurally, it's similar to:

That is, generally, you can have four layers of memory, for example:

Layer Content Saved Typical Storage Lifecycle
Working Memory Current task steps, temporary variables, recent messages Graph State / Checkpoint Current thread
Summary Memory Project goals, key decisions, unfinished items Summary table or thread summary Medium-term
Semantic Memory Historical conversation snippets, code discussions, events Vector DB Long-term, retrievable
Structured Memory Stable preferences, project configs, contacts, versions Relational DB / KV / Document DB Long-term, updatable

Simply put:

These things need to be stored separately and cannot be mixed into a single vector database.

In reality, memory writing also needs a strategy. From a user's perspective, not every sentence is worth long-term preservation. So, before writing, at least judge:

For example, after a project switches from Flutter to Swift, old memories can no longer be recalled under the "current tech stack." Therefore, based on the business form, we need to maintain version relationships: used Flutter in the past, a decision was made on a certain day to switch to Swift, and the current valid value is Swift.

Finally, it's also necessary to prevent memory pollution, such as content read from web pages, emails, or tool results that might contain malicious instructions.

Therefore, external content can only serve as data and cannot be automatically upgraded to system instructions or long-term preferences. It is generally recommended to save structured information for memories, such as:

Source, creation time, confirmation status, credibility, validity period, override relationships, and sensitivity level. Important facts are best written after programmatic verification or user confirmation.

For this part, you can also look at the introduction in "Claude Opus 5 System Prompt Leak". Claude designed a set of persistent memory filesystem specifications within its system prompt.

7. Evaluation

Evaluation is also a key implementation. In Agent development, the entire evaluation mainly looks at the execution trajectory:

Generally, a relatively complete evaluation can be divided into four levels, for example:

So, building an Agent might be fast, but running evaluations, testing stability, reliability, and model compatibility is actually the most time-consuming and labor-intensive process.

Then, the evaluation for RAG must also be able to trace back to the fault layer. Because in a "Contract Assistant," RAG problems are the biggest and most likely to occur. So, you need to know the trajectory of RAG problems. Trace information must support tracing back to the fault layer, for example:

Phenomenon More Likely Problem Priority Investigation
Low Context Recall Key materials not retrieved Chunk, Query Rewrite, Hybrid Search, Top-K
Low Context Precision Too much noise Reranker, Metadata, Deduplication, Query Understanding
High Recall but Low Faithfulness Materials found, model still hallucinating Generation Model, Prompt, Citation Constraints, Context Conflict
High Faithfulness but Low Answer Relevancy Not hallucinating but answering the wrong question Intent, Question Rewriting, Prompt, and Completion Conditions
Tool call succeeds but task fails Tool result did not solve user goal Result Validation and Re-planning
High offline score, low online satisfaction Data distribution or business goal mismatch Online Samples, Task Definition, Product Interaction

The most important thing is to establish a complete Agent test set.

8. Observability Trace

Trace is the most important entry point for AI debugging of an Agent. We talked a lot about trajectories earlier, and that refers to Trace. An Agent must record a complete trajectory.

For example, record for each run:

trace_id / task_id / thread_id
User and tenant context
Model, version, temperature, token usage
Routing result, confidence, and risk level
State snapshots and node latency
Recall Query, candidate Chunks, reranking scores, and citations
Tool candidates, final selection, parameters, and Schema version
Authorization, confirmation, and policy decisions
Tool latency, status codes, retries, and idempotency keys
Final completion conditions and failure reasons
Manual modifications and user feedback

Of course, in logging, we cannot save Prompts, email bodies, and sensitive parameters without boundaries. Trace itself must also adhere to minimization, desensitization, access control, and retention periods. For example, key dashboards can include:

In short, the Trace must be complete. As for whether desensitization is needed, that actually depends on your business.

Conclusion

So, these are all the basic Agent logical concepts, and also an introduction to the Agent capabilities needed for a "Contract Assistant." Whether you build it the old-fashioned way or with AI, you need to plan holistically on a similar foundation. The core is how to build a Flow and a golden test framework based on your business.

The quality and effectiveness of an Agent must have a set of quantifiable metrics. The entire goal and function of an Agent is to make AI reliable and controllable. In a coding Agent, a hallucination might not be a big problem, but in contract, legal, or shopping scenarios, a single hallucination or escaped operation can lead to a huge P0 issue.