跪拜 Guibai
← All articles
Agent

From ChatGPT to Production-Grade Agents: A Complete Engineering Map

By J船长 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

The gap between a demo agent and a production system is mostly not model intelligence; it's the mundane machinery of permissions, state recovery, idempotency, eval, and observability. This series maps all that machinery onto a single realistic business case, giving Western engineers a practical blueprint for shipping agents that don't just impress in a demo but survive real users, restarts, and adversarial inputs.

Summary

The first post in the Agent Development series lays out a complete engineering framework for moving from a plain LLM chat interface to a production-grade agent system. It uses a concrete scenario: a user says an order in Google Play Billing was paid, but coins never arrived. A bare LLM can only guess at causes from general knowledge; an agent can query the order system, check grant records, retrieve internal policy documents, and — when reissuance is needed — request human approval before writing anything.

The post walks through the core mechanics: Function Calling lets the model return a structured tool request, the Agent Runtime executes it and feeds results back, and the Agent Loop keeps the cycle going. It then categorizes architectural patterns—Workflow, ReAct, Planner-Executor, Multi-Agent, and Human-in-the-Loop—making clear that higher model autonomy is not always better. For fixed rules like payment validation, a deterministic workflow beats a free-wheeling model.

Beyond the basics, the article covers the less glamorous but critical infrastructure: context engineering and memory, state persistence with checkpoints, idempotency to prevent duplicate grants, a model gateway for routing and cost control, eval with golden datasets and LLM-as-Judge, observability with traces and spans, and security mechanisms like prompt injection defense, least privilege, and approval binding. The final sections tie it together into a reference architecture and even include a two-minute interview explanation. A planned 12 follow-up posts will build these mechanisms incrementally in a runnable teaching project.

Takeaways
— An LLM is a model; an Agent is a system that uses models plus tools, an execution loop, and control mechanisms to achieve tasks.
— Function Calling converts a natural language request into a structured tool call (e.g., {name: get_order_status, arguments: {order_id: 1001}}), which the backend validates and executes.
— The Agent Loop is the cycle of model decision → tool execution → result feedback → next decision, organized by the Agent Runtime.
— Higher model autonomy doesn't mean a better system; rule-defined flows like payments should use Workflow, while open-ended troubleshooting suits ReAct.
— ReAct focuses on task strategy (how to decide next based on observations), while Agent Loop focuses on execution mechanics; one can exist without the other.
— Context Engineering provides the most relevant, trustworthy, and necessary info within token budget, not just more context.
— State is backend-maintained task progress for execution/recovery; Context is what the model sees for decisions. They are related but not identical.
— RAG uses vector retrieval (Embedding) and keyword retrieval (BM25), optionally combined as Hybrid Search and refined by Re-ranking.
— Checkpoints save task state and execution position for recovery after crashes; idempotency keys (e.g., grant-request-888) prevent duplicate business effects on retry.
— A Model Gateway centralizes model routing, rate limiting, retries, fallback, and cost tracking across multiple agents.
— LLM Eval uses Golden Datasets, deterministic assertions, and LLM-as-Judge to verify not just answers but tool calls and business outcomes.
— Observability records traces and spans so you can locate whether an error came from model input, tool execution, context feedback, or final generation.
— Agent security must handle prompt injection via documents, enforce least privilege at the tool and data level, and bind human approvals to specific operation parameters.
— Production readiness has three stages: run a single task, handle failures and restarts, and sustain safe modification with regression eval and rollback plans.
— AGENTS.md, SDD, and Harness each play distinct roles: project context for coding agents, specification-driven development, and runtime/control mechanisms.
— An interview answer should cover what the system does, how it executes, how it handles risk, and how it validates results, using one concrete case.
— The full architecture integrates context management, state/checkpoint storage, model gateway, tool dispatch, security, eval, and observability around the Agent Runtime.
Conclusions

The biggest takeaway is that a production agent's value lies in the boring parts: permission checks, state persistence, idempotent writes, and replayable eval. No amount of prompt cleverness compensates for a system that can't safely resume after a crash or refuses unauthorized grants.

The article wisely avoids prescribing a single architecture. It shows that the same agent can switch between Workflow, ReAct, and Planner-Executor per task, which aligns with how real systems should be designed rather than forcing one pattern.

The emphasis on separating model decisions from system controls (e.g., recording an unauthorized model proposal even if the backend blocks it) is a sober way to evaluate agent quality: the final outcome isn't enough; the intent matters for debugging and safety.

Checkpoints alone aren't enough; the article correctly pairs them with idempotency at the business system level. This is the kind of detail that separates demos from production, and it's often missing in beginner tutorials.

The interview section is practical but revealing: the expected answer is not about model capabilities but about engineering responsibility chains — who checks what, based on what evidence, and what happens on failure.

Concepts & terms
Agent Runtime
The component in an agent system that organizes the execution loop: sending messages to the LLM, receiving function-call requests, dispatching tools, returning results, and deciding whether to continue, wait, or stop.
Function Calling
A protocol where the LLM returns a structured request specifying a tool name and arguments, allowing the backend to validate and execute the intended operation. It bridges natural language and executable business APIs.
Agent Loop
The iterative cycle of model decision, tool execution, result feedback, and another decision that continues until a task goal is met or a budget/approval condition stops it.
ReAct
A reasoning-on-action pattern where the agent alternates between thinking about what to do, taking an action (e.g., calling a tool), observing the result, and adjusting the next step. Good for open-ended exploration and troubleshooting.
Checkpoint
A persisted record of an agent task's state and execution position at a specific moment, enabling the system to resume after a crash or restart. Must be stored outside process memory.
Idempotency (key)
A property where repeating the same business operation produces the same result as doing it once. An idempotency key is a stable identifier sent with a write operation; the receiving system uses it to deduplicate requests, preventing double issuance (e.g., double coin grants).
Model Gateway
A unified access layer that sits between agent runtimes and LLM APIs, handling model selection, routing, credential management, rate limiting, retries, fallback, and usage/cost recording.
LLM-as-Judge
Using a separate LLM to evaluate an agent's answer against criteria and evidence. It's a cheap way to score semantic quality, but needs calibration and human oversight because judges can be biased (e.g., favor longer answers).
Golden Dataset
A human-curated set of test cases with expected behaviors and pass conditions, used to run regression evals on agent changes. It should cover both normal and abnormal scenarios.
Trace and Span
A Trace is the full execution chain of a task; a Span is a single operation (e.g., one model call or tool call) within that chain. Together they let you follow a request through all components for troubleshooting.
Prompt Injection (indirect)
An attack where malicious instructions hidden in retrieved documents, web content, or other external inputs manipulate the LLM into requesting unauthorized actions. Defense requires separate context labeling and strict execution-layer permission checks.
Hybrid Search and Re-ranking
Hybrid search merges vector (semantic) retrieval and BM25 (keyword) retrieval for broader recall; re-ranking then re-scores the merged candidates to select the most relevant passages before feeding the LLM.
Context Engineering
The practice of assembling the most relevant and trustworthy information for a model call within token budget, including instructions, history, tool schemas, results, and retrieved knowledge, to improve output quality without overflowing the window.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗