跪拜 Guibai
← All articles
Agent · Backend · Architecture

How Huolala Turns AI Agents into Real Business Results: Three Production Deployments and a Four-Layer Model

By 货拉拉技术 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

This is a rare, production-level account of AI agents operating inside core transaction systems, not just demos. It shows where agents actually add value, what constraints must live in code rather than prompts, and how to define success with business metrics while controlling token costs.

Summary

Huolala has shipped AI agents into its real-time freight transaction systems, not as chat assistants but as agents that directly change business state. The engineering leader behind this work, Laurence Qi, walks through three live deployments: AI-guided ordering that turns open user intent into confirmed orders, a continuous matching system that keeps pushing toward a deal after an order goes unanswered, and an operations agent that executes strategies across internal systems. In each case, agents work alongside existing ML, operations research, and transaction systems rather than replacing them.

A central piece is the four-layer model for productionizing agents. Model capability is the common starting point; the harness determines the floor, with two distinct kinds of constraints. Help-the-model harnesses compensate for current model weaknesses and should be deletable as models improve. Protect-the-business harnesses, like permission checks, idempotency, and state machine SOPs, must be written in code and never disappear. A dual loop separates agent capability metrics from business outcome metrics, showing that improving dialogue quality alone can miss the real bottleneck, such as targeting the wrong driver pool. The ceiling comes from how well enterprise knowledge is encoded: deterministic rules into code, quality standards into evaluation, and effective scripts into strategy.

Measured results include 99% intent recognition accuracy, 93% vehicle selection accuracy, 25% faster ordering for new customers, a 12% relative match rate improvement with 4x ROI versus manual matching, and an 80% efficiency gain from 33+ operations agents with zero configuration errors.

Takeaways
— AI agents at Huolala extend system capability boundaries, not redo existing ML, optimization, or transaction systems.
— Intent ordering reached 99% intent recognition accuracy and 93% vehicle selection accuracy; new-customer order time dropped 25%.
— Continuous matching improved match rate by 12% relative and achieved 4x ROI compared to manual methods.
— Over 35% of operations effort was spent on manual execution; 33+ operations agents improved execution efficiency by 80% and cut configuration errors to zero.
— The four-layer model: model sets the starting point, harness sets the floor, dual loop sets direction, business knowledge sets the ceiling.
— Two harness types: help-the-model harnesses should be designed to be deletable; protect-the-business harnesses (SOP, price, permissions, idempotency, audit) must be code.
— The dual loop separates inner-loop agent capability metrics from outer-loop business metrics; bad cases must be defined by business attribution.
— A fully agentic matching approach failed to follow SOPs; a high-code harness with state machines fixed it.
— Agents do not replace algorithms; they organize discrete decisions into continuous action.
— The company currently keeps business decisions with humans; agents execute within authorized boundaries.
— LLM-as-a-Judge is used to scale evaluation, but it is calibrated by human spot checks and validated against business metrics.
Conclusions

Model capability is no longer the differentiator; the harness and the way business knowledge is encoded decide whether an agent helps or hurts.

Improving agent dialogue quality while ignoring which drivers are targeted can actually drive conversion down; the outer loop catches what the inner loop cannot see.

The cautious stance on letting agents set strategy reflects a practical trade-off: the deeper agents go into execution, the more guarded they become near decisions.

Deletable help-the-model harnesses indicate that agent production is still evolving rapidly, while protect-the-business harnesses will remain a permanent engineering layer.

Concepts & terms
Harness Engineering
Designing the code and constraints that wrap an LLM to ensure it behaves safely and usefully. Two types: help-the-model harnesses compensate for model weaknesses and should be removable as models improve; protect-the-business harnesses enforce business rules and must be permanent code.
Dual Loop
A feedback structure with an inner loop measuring agent capability (intent recognition, instruction following, bad-case rate) and an outer loop measuring business outcomes (conversion rate, ROI, efficiency). The outer loop defines what the inner loop should optimize.
LLM-as-a-Judge
Using a large language model to evaluate another model's outputs at scale against expert criteria, with human spot checks to calibrate the judge's own bias.
Gold Script Extraction
Analyzing high-performing customer interactions to extract effective dialog strategies, then feeding those strategies back into the agent to improve future performance.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗