How Huolala Turns AI Agents into Real Business Results: Three Production Deployments and a Four-Layer Model
How AI Agents Enter Real-Time Transaction Systems: Three Practices and an Engineering Methodology from Huolala
Author: Qi Longji (Laurence Qi), Technology Director at Huolala
Abstract: When AI agents move from "answering questions" to "directly changing business state," the real challenge is no longer whether the model can think, but whether it can act reliably in real-time transaction systems and deliver measurable business results. This article is based on Huolala's production practice across three scenarios: intent trading (AI ordering), continuous matching (driver agent / AI matching), and strategy execution (AI operations). It discusses how agents share responsibilities with mature machine learning, operations research optimization, and transaction systems, and proposes an engineering methodology for production systems: the four-layer model of agent productionization. The model determines the starting point, the harness determines the floor, the dual loop determines direction, and business knowledge determines the ceiling.
Keywords: AI Agent; Real-Time Transaction Systems; Harness Engineering; Context Engineering; Multi-Agent; Evaluation Loop
Introduction
Over the past three years, the industry has invested significant effort in getting agents to actually complete tasks: stronger models, longer contexts, more tools. But for a business company, one question has become increasingly difficult to answer: with all this token consumption, what new business value has actually been created?
At QCon, I conducted a small survey: almost everyone raised their hand when asked who was working on agents, but significantly fewer could clearly state which business metric improved because of an agent. Huolala went through the same phase. The real difficulty isn't building an agent; it's articulating what it has expanded.
Our later conclusion is not complicated: use agents to expand the boundaries of system capabilities, not to redo existing capabilities. Below, I first explain how this conclusion formed, then use three production practices as evidence, and finally present the engineering methodology I derived from them: the four-layer model of agent productionization.
1. The Value of an Agent Is Not in Redoing Existing Systems
To answer "what new value does an agent create," you first need to see what kind of system it is entering. From ChatGPT to coding agents to long-running agents that operate for dozens of hours, AI has begun to enter core business chains. The application boundary has also moved: from employee efficiency, business assistance, to business execution, and even direct participation in transactions. The further along this path, the higher the value but also the greater the risk. When assisting, humans still control decisions. Once an agent directly changes business state, hallucinations and errors become real losses.
So the focus shifts. Previously, the focus was on "can it answer correctly, can it plan correctly"; after entering core business, you must also ask: can it reliably complete the task?
Reliability is only a prerequisite. After repeated attempts, we started to invert the question: if we don't use an agent, could this task have been done anyway? What scenarios genuinely deserve an agent? We converged on three criteria:
| Criterion | Question |
|---|---|
| New capability | After using the agent, what new capability has been added that wasn't possible before? |
| New result | Does it bring new, incremental business results? |
| Cost coverage | Can the benefits cover token costs and the system complexity introduced by the agent? |
In one sentence: if you can solve it without an agent, first ask why you still need an agent.
Applying this criterion to existing systems, the boundary is actually very clear. Inside the boundary are ML prediction, operations research optimization, business rules, and transaction and fulfillment systems. These capabilities are mature and should continue to be handled by them. Outside the boundary are open-ended tasks that require understanding, planning, coordination, and adjusting to on-site conditions: users need to express requirements and make ordering decisions themselves, drivers need to refresh and grab orders and plan their day, and operations staff need to configure strategies across multiple systems and launch them. Previously, these tasks could only be done by humans.
Therefore, agents should not replace what existing systems already do well. Prediction goes to ML, transaction processing goes to engineering systems. What agents should take on is the open-ended work that only humans could do before, converting it into system capabilities that can sustainably achieve goals through understanding, planning, and tool calls.
Figure 1: Use agents to expand system capability boundaries, not redo existing capabilities
For Huolala, this judgment has clear landing points. The real-time transaction chain has been built for over a decade; pricing, ordering, matching, acceptance, fulfillment, and completion are mature. We are not rewriting this chain with agents but expanding on it with three types of capabilities: intent trading in pricing and ordering, continuous matching in matching and acceptance, and strategy execution across the entire chain. Details follow.
2. Intent Trading: From Expressing Intent to Completing an Order
For ordinary users outside the logistics industry, completing a freight order is often not about operations but professional judgment. You need to distinguish between hauling goods vs. moving house, full truckload vs. LTL. For vehicle types, even vans alone have micro, small, medium, and large sizes, plus additional requirements like refrigeration, tailgate, and side opening. What users really want is just "get the goods moved," but accurately expressing intent into an executable order still requires some freight knowledge.
If you only replace form filling with natural language interaction, it becomes faster, but the professional threshold doesn't disappear. Users still need to know how to describe their needs and choose the right service. What we truly want is to let users avoid having to become half a logistics expert: using intent understanding and domain expertise to directly complete intent-based ordering.
Turning an open-ended intent into a real transaction isn't just about dialogue. The division of system responsibilities must be clear. Users express needs and confirm key information like location and price; agents understand intent, record confirmed information, and ask follow-up questions at critical points; transaction facts must come from authoritative transaction systems. Models can hallucinate, and user descriptions can be inaccurate: addresses need secondary confirmation, vehicle types vary by city, and prices can only come from the pricing system. In other words, agents organize and understand existing capabilities; mature systems verify and execute.
Once the division is clear, the next question arises: models are inherently uncertain, so how do we make them reliably execute transactions? Our approach can be summarized in three aspects.
First, define boundaries. Faster, more accurate, and more generic cannot all hold at once. In ToC scenarios, response time is almost non-negotiable: model inference already takes time, and if too slow, retention and conversion suffer directly. So you must narrow the task scope and choose models based on difficulty: small models for simple tasks, larger models for deep-reasoning tasks like vehicle selection.
After boundaries are set, context must be managed effectively. Initially, we passed the full dialogue history directly to the model. When users modified a requirement or location, the model often referenced wrong information. We switched to a structured order draft: what the goods are and whether they are confirmed; loading location and whether it has been validated; current price and order status. When information changes, we re-confirm with the user rather than letting the model infer from a long history.
Even so, open understanding cannot directly translate into a transaction action. We need deterministic guardrails: transaction facts like address and price are determined by the system; price and order are executed only after user confirmation; actions are constrained by permissions, state machines, and idempotency. These three aspects together can be summarized as: let the model be uncertain only where uncertainty is worthwhile.
At this point, the transaction chain has a basic closed loop. But for intent-based ordering to be truly usable, one key capability is still missing: professional vehicle selection. When a user says "help me move a double-door refrigerator," they often don't know its size, which vehicle can fit it, or whether it should be transported upright. Previously, they had to search on their own or ask customer service or drivers. Now agents, domain rules, and transaction systems work together: the LLM identifies the category and asks for dimensions and special requirements; validated loading rules are encoded as weight, dimension, and placement constraints in code; the system compares the load capacity and cargo box dimensions of available vehicle types in the city, eliminates options that can't fit, and explains the recommendation reason to the user. If information is insufficient, it keeps asking; it never guesses. Understanding goes to the model, constraints go to code, facts go to the vehicle model library.
When professional knowledge enters the transaction process, business results change: intent recognition accuracy reached 99%, vehicle selection accuracy 93%, and new-customer ordering time was shortened by 25% compared to traditional methods. The value of agents is not just understanding users but also turning previously human-dependent experience into systems and measurable returns.
3. Continuous Matching: After a Matching Failure, How the System Keeps Pushing Forward
Ordering solves "how intent becomes an order"; after an order is generated, the real difficulty is often matching, especially whether the system can continue toward the goal after a failed match.
Look first from the driver side. Freight drivers typically grab orders by refreshing a list, finding orders they want to accept, and then grabbing them. But drivers care less about "what the next order is" and more about "how to arrange this day" - for example, income is still 300 yuan short, ideally heading home on the way, and finishing before 10 PM. Previously, this goal depended on the driver repeatedly screening and judging. With agents, the continuous task of finding orders can be delegated along with the goal. This change isn't just an extra assistant; it's a paradigm shift: from one-time recommendations to goal-oriented continuous service, from Recommendation to Delegation.
The order side faces a similar gap. If an order remains unrouted for a few minutes, it enters the long tail. Previously, the approach was mainly to expand broadcasting and then wait for users and drivers to contact each other. Now the system can continuously push toward closing the deal: observe the state, choose tools, and proactively contact both parties using channels like AI outbound calls. This process has a clear loop: Observe → Decide → Act → Feedback → Replan, and terminates when a successful match, state change, timer expiry, or risk trigger occurs. Continuous action is valuable, but continuous action without a termination condition is itself a risk.
To bring this continuous push into production, we adopted a Multi-Agent architecture. The key isn't the number of agents but how boundaries are divided: in the middle is the matching brain, which judges current strategies and tools based on business goals; on both sides are state/context management and deterministic harnesses like permissions, tool contracts, idempotency, timeouts, auditing, and human takeover; below are execution agents like driver calling and user calling, plus channels like RTC, IM, Push; underneath are still recommendation, prediction, operations research, pricing, and transaction systems. When the LLM context gets too long, attention scatters; an agent that takes on too many responsibilities tends to perform worse. Making each agent's loop small makes them easier to evolve independently.
After the boundaries are clear, a common question arises: aren't the original algorithms still important? On the contrary, they are even more important. ML is good at problems that LLMs are not; LLMs are good at thinking, planning, calling tools, and deciding next steps based on feedback. So the division is clear: ML/recommendation handles acceptance probability and candidate ranking; OR/pricing handles constrained optimization and resource allocation; rules and transaction systems hold hard constraints and facts; agents judge timing, choose calls, execute actions, read feedback, and adjust. Agents don't replace algorithms; they organize discrete decisions into continuous action.
For a typical long-tailed order, the path is: first use models like acceptance probability to find more suitable drivers, then choose outbound calls, IM, or video/audio to reach out, collect willingness and quotes, negotiate price and feedback with the user at the right time, and finally push to a deal. What really matters is not whether the number of touches is sufficient, but whether the feedback from both sides can be aggregated into a verifiable negotiation plan.
Since continuous matching went live nearly a year ago, match rate has improved by 12% relative, and ROI is 4 times that of manual methods. The latter number is especially important: as token consumption increases, the business side will inevitably focus on cost. Looking only at effectiveness without cost can easily produce a system that is very effective but not worth it.
4. Strategy Execution: Humans Set Strategy, Agents Operate the System
If the first two practices expand the capability boundary on the user and driver sides, there is also a long-underestimated open-ended task on the operations side: after a strategy is determined, how to execute it reliably.
Internal statistics show that over 35% of operations effort is spent on manual execution. The strategy may already be clear, but landing it still requires data collection, cross-system targeting and configuration, and post-release verification item by item. Much of this is mechanical labor. Humans have to both judge and operate, and the latter becomes the biggest drain.
So the division is redefined: humans set goals, boundaries, and authorization; agents understand, break down, and call tools; business systems handle final actions and facts. The core of an operations agent is a stateful execution runtime that continuously performs goal understanding, task planning, tool calling, and result verification. Input is business goals and strategies; output is verifiable execution results.
Because the output genuinely rewrites business state, one rule is non-negotiable: all business actions must go through standard business tools to connect to real systems. LLMs hallucinate, and operations actions have real impact; a critical configuration could even affect the entire business. Therefore the harness layer must handle parameter validation, boundary and permission control, auditing, and human feedback. Evaluation no longer looks at dialogue quality but at task completion rate, execution accuracy, efficiency, business impact, and human intervention rate. For operations staff, the way of working becomes: express an intent, the agent breaks it into steps and executes them step by step, and areas requiring confirmation enter an approval flow. What used to be item-by-item manipulation becomes a traceable, approvable landing chain.
Once execution becomes reliable, another question almost always comes up: if agents are so powerful, can they directly decide business strategy? Our current answer is no, at least not at this stage.
| Level | Division of labor |
|---|---|
| Standardized execution | Agent executes within authorized boundaries |
| Exception handling | Agent identifies and escalates; humans review |
| Strategy suggestion | Agent assists; humans choose |
| Business decisions | Humans are responsible for goals, budgets, risks |
From standardized execution to business decisions, responsibility and impact escalate. Letting an agent directly decide overall operational strategy would have a massive impact on the business. So our principle is clear: the deeper into execution, the more cautious toward decision-making. First make execution reliable, then discuss greater autonomy.
Following this boundary, we have completed more than 33 operations agents. Using manual execution efficiency as baseline, execution efficiency improved by 80%; within the measured scope, manual operation errors such as configuration mistakes dropped to zero. The next step is to cautiously expand to process and decision assistance, only after reliable execution is established.
5. The Four-Layer Model of Agent Productionization
The three practices have different users, goals, and interaction modes, but looking back, the same set of problems recurs: how can an uncertain model act reliably? Does better capability equal better business? Where should enterprise private knowledge reside in the system? Based on these common challenges from production practice, I propose the four-layer model of agent productionization as an engineering methodology for taking agents from demo to production.
Figure 2: Model determines starting point, Harness determines floor, Dual Loop determines direction, Business knowledge determines ceiling
| Layer | Role | Principle |
|---|---|---|
| Model | Starting point | Models are a common starting point for all teams |
| Harness | Floor | Help-the-model harness must be designed to be deletable; protect-the-business harness must be code |
| Dual Loop | Direction | Outer loop decides what inner loop optimizes; bad cases are defined by business attribution |
| Business knowledge | Ceiling | Deterministic knowledge goes to code; standards go to evaluation; effective plays go to strategy |
Capability across models is obtained almost simultaneously; the real differentiator is the next three layers. The harness determines whether a system can go online. If the worst-case outcome is unacceptable, the system cannot enter production. The dual loop determines the direction of continuous improvement. Business knowledge determines how deep the agent can ultimately go into the business. Let's expand in this order.
Harness Determines the Floor: First Distinguish Two Types of Constraints
Look at the floor first. There are two views in the industry: one says the stronger the model, the more the harness will eventually disappear; the other says the harness will never disappear. I think both are right because they are talking about different types of constraints.
Help-the-model harness compensates for the model's current deficiencies: task decomposition, context assembly, format constraints, compensatory retries. A few years ago, models struggled to output stable JSON, so such constraints were heavy; as models get stronger, this layer becomes thinner and must be designed to be deletable. Protect-the-business harness protects the business itself: whether SOPs are followed, whether prices are transaction facts, permissions/idempotency/auditing, human takeover. It is independent of model strength; it defines the boundary between the model and the real business system and determines the scope within which the model can change business. This kind of constraint will not disappear and must be written in code.
A simple way to judge: if the model were already perfect, would this constraint still be necessary? If yes, it is a protect-the-business constraint and must go into code. In our three scenarios, these are: address and vehicle-type validation in ordering, the state machine that carries matching SOP in matching, and parameter validation, permissions, idempotency, auditing, and approval for standard tools in operations.
After distinguishing the two types, the harness also has a subtler role: putting certainty in the right place. The evolution of AI matching illustrates this best. In the first phase, it was fully agentic: the model autonomously planned how to match and communicate, resulting in difficulty following SOPs and unmeasurable processes, making it impossible to judge whether a strategy worked. In the second phase, workflows orchestrated large nodes and state machines constrained the model's internal state; SOP following improved, but context between nodes became redundant and couldn't be supplied on demand. The third phase moved to a high-code harness: using code to orchestrate LLM nodes and state machines, controlling SOP and assembling context on demand. The primary value of deterministic structures is making the business process measurable; only when the process is measurable can strategies be optimized.
This also means that in core businesses, more autonomy is not better. First make the process followable and measurable, then discuss expanding autonomy. If help-the-model constraints are coupled with protect-the-business constraints from the start, they become hard to cleanly remove even when the model improves. "Deletability" must be designed in advance.
Dual Loop Determines Direction: The Outer Loop Sees Problems the Inner Loop Cannot
The harness solves "can it go online and can the process be measured." Once measurable, the next question is: which direction should optimization move? We split this into two loops.
The inner loop is the agent capability loop: intent recognition, instruction following, bad-case rate, the work of doing things right. The outer loop is the business metric loop: primary and secondary business metrics, which determine what the right things are. Both are necessary, but they have different priorities.
For a period, the inner loop kept improving while match conversion declined. The root cause wasn't dialogue ability; it was not targeting drivers with higher intent. No matter how strong the conversation is, if the contact target is wrong, conversion is hard to achieve. Such problems are invisible if you only watch the inner loop. They must be discovered by the outer loop, and bad cases must be defined by business attribution. Once direction is correct, strengthening the inner loop capability will yield business results.
Once direction is right, loop speed determines the iteration rate of capability. We use the LLM itself to accelerate two things. One is LLM-as-a-Judge: evaluating dialogues and tasks at scale against expert standards, producing bad-case distributions, then optimizing prompts and strategies, while using manual spot checks to calibrate the judge. The other is gold-script extraction: analyzing effective strategies from high-performing phone calls and feeding them back, then validating against outer-loop business metrics. The judge is not final: humans calibrate it, and business metrics validate it. The evaluation system itself also needs to be evaluated: a biased judge systematically biases optimization direction.
Business Knowledge Determines the Ceiling: Route by Determinism
Even if the harness holds the floor and the dual loop calibrates direction, how far an agent can go in the business ultimately depends on how much enterprise private knowledge it masters. Loading fit, transport experience, understanding of people/vehicles/goods/route networks, whether a strategy works, these are not in public corpora and LLMs don't have them by default. Only after engineering them does an agent gain the ability to significantly impact the business.
The engineering principle is to route by determinism:
| Knowledge type | Destination | Example |
|---|---|---|
| Deterministic knowledge | Code and harness | Vehicle model library and loading constraints, matching SOP state machine |
| "What is good" | Evaluation and metrics | Experts define evaluation criteria, label bad cases |
| Effective plays | Strategy and scripts | Gold scripts extracted into strategies, fed back to agents |
Writing into a prompt is not knowledge engineering. The industry often says "LLM + Harness = Agent"; in core business, you need to add another item: business knowledge. Business experts are the agent's teacher and judge: they teach what works and define what is good; product and engineering provide the corresponding capabilities. The quality of knowledge jointly extracted by business, product, and engineering determines the ceiling.
The Same Methodology on Three Business Lines
Put the three principles back into the three practices, and you'll see the same methodology landing on different business lines:
Figure 3: The same principles land on three different agent types
| Principle | Intent Trading: AI Ordering | Continuous Matching: AI Matching | Strategy Execution: AI Operations |
|---|---|---|---|
| Harness | Structured order draft + deterministic guardrails | High-code harness: code orchestrates LLM nodes and state machines | Business actions only through standard tools |
| Dual Loop | Inner: intent recognition, vehicle selection accuracy; outer: order time | Inner: intent recognition, instruction following, bad-case rate; outer: match conversion, match rate, ROI | Inner: task completion, execution accuracy; outer: execution efficiency, business impact |
| Business knowledge | Loading rules -> code constraints | SOP -> state machine; scripts -> strategy | Operations SOP and rule boundaries -> tools and approval |
Different users, goals, and interactions, but the principles land individually. That's why I think it should be called a "methodology" rather than "a summary of project experience."
Finally, I need to clarify the scope of applicability. The four-layer model targets core business scenarios where existing systems hold business facts. If the facts themselves are generated by the agent, or no mature system exists to rely on, the meaning of "protect-the-business harness" and "transaction fact authority" will change, and the principles need to be re-derived.
Conclusion
Back to the initial question: with all this token consumption, what new business value has been created? For us, the answer isn't that we launched several more agents, but that the boundary of system capability has been pushed outward: open-ended tasks that previously could only be done by humans are starting to become measurable, continuously improvable system capabilities.
AI is advancing fast, and many technical people feel anxious. I tend to be optimistic: what agents extend is not just model capability, but the boundary at which technology can solve business problems, which is the boundary of what technical people can solve.
Model determines the starting point, harness determines the floor, dual loop determines direction, business knowledge determines the ceiling. I hope this four-layer model helps readers who are bringing agents into production.
About the Author
Qi Longji (Laurence Qi) is Technology Director at Huolala, responsible for core transaction chains, freight middle platform, and freight and enterprise businesses. He was previously a senior architect at Ele.me and Alibaba Local Life Services, and led Ele.me's logistics to upgrade from a million to ten million daily orders. He is currently focused on applying AI agents in business scenarios.