Dewu's 'TuiChaCha' Agent Turns Recommendation-System Debugging Into a Self-Learning Assembly Line

This article is a technical record compiled from a presentation by Dewu technical experts at AICon Shanghai.

It is the middle installment of the 'Dewu Recommendation AI Harness Engineering Practice Series,' a three-part serial. This installment (the middle) further details the complete technical system for AI code generation, protection verification, and safe deployment in Dewu's complex recommendation business scenarios. It covers the overall architecture framework of the self-developed AI Harness, the full-process safety protection mechanism, the core algorithm implementation of the hybrid agent, and the practical details of industrial-grade engineering implementation.

1. Past and Present: From 'Manual Troubleshooting' to 'Systematic Accumulation'

'A notification sound rings, and an abnormal recommendation case alert is pushed. On the screen is an anomalous recommendation case; behind the screen, developers face the thorny problem of massive trace logs. Recall, coarse ranking, fine ranking, strategy filtering... In this multi-layered logical maze, every filter_code or abnormal score is like a puzzle to be solved. In the past, we relied heavily on individual experience, even requiring multi-party consultations, to dig out the hidden 'landmine' from the dense code. If logical coupling or experimental conflicts were involved, the troubleshooting process was no different from a long marathon, which was not only inefficient but also trapped the team in low-efficiency, repetitive manual labor. Under this deep black-box operational pressure, developers were always passively handling various online emergencies.'

Now, the emergence of 'TuiChaCha' is ending this era of blind troubleshooting. It is the 'autonomous driving diagnostic expert' in the recommendation system domain, making complex algorithmic logic transparent, traceable, and predictable. From today onwards, say goodbye to late-night anxiety and let the recommendation system truly enter the 'white-box era'! The anxiety of recommendation system troubleshooting is, in essence, knowledge asymmetry and information fragmentation.

1.0 Experience-Driven Era (Early Stage): This phase heavily relied on static documentation and the personal experience of core members. When anomalies occurred, the team often needed to gather information from multiple sources, manually search logs, and grab keywords to piece together context. This 'carpet-style' search method was extremely inefficient, and the troubleshooting process was like a blind gamble.
2.0 Platform Tool Era (Mid-Stage): With the integration of APIs for screenshot recognition, log querying, and profile dashboards, we digitized the troubleshooting process. Although this solved the problem of 'being able to see,' it was still essentially 'human-driven machines.' Massive diagnostic experience could not be accumulated, and the troubleshooting chain still heavily relied on expert judgment, making it difficult to form a standardized closed loop.

Breaking the Ceiling: The 'Three Cognitive Dilemmas' of Traditional Recommendation Diagnostic Solutions

In the exploration of automated troubleshooting, we have implemented various technical solutions and summarized three typical limitations common in traditional architectures:

Limitation 1: The 'Linear Rigidity' of SOP Scripting. This approach attempts to solidify expert paths into fixed scripts. While it can quickly respond to known issues, the drawbacks are obvious: the rapid iteration of recommendation strategies and underlying architecture means the script update speed always lags behind business changes. Faced with 'new question types,' such inflexible solutions often exhibit high maintenance thresholds and very weak generalization capabilities.
Limitation 2: The 'Efficiency Gap' of the Pure Agent Model. Directly using large language models (LLMs) for end-to-end autonomous troubleshooting still faces severe challenges at this stage. Excessively high inference costs (token consumption) and long response latency (minute-level waiting) make it difficult to meet the stringent SLA requirements of a production environment. Furthermore, the potential logical deviations (hallucinations) of the model itself add to the time cost of secondary verification.
Limitation 3: 'Knowledge Loss' from Experience Silos. Due to the lack of an effective accumulation mechanism, core diagnostic logic often resides in personal experience. When personnel change, a serious 'knowledge gap' occurs. Existing tools mostly remain at the level of 'displaying phenomena' (outputting status codes) and lack the ability to 'assist decision-making' (outputting action paths). The disconnect between diagnosis and action keeps the Mean Time To Repair (MTTR) consistently high.

2. The Solution: 'TuiChaCha' Takes the Stage

In the early construction phase of 'TuiChaCha,' we deeply weighed the options: customize based on a general open-source framework, or adopt the more engineering-certain OpenClaw? Ultimately, we chose the latter, aiming to bridge the 'last mile' of landing intelligent diagnosis in a production environment:

From 'Process Coupling' to 'Atomic Plugins': Traditional chain calls, when dealing with the complex logical nesting of a recommendation system, easily fall into unmaintainable logical coupling. OpenClaw's plugin specification (Plugin/Skill) allows us to encapsulate APIs from various platforms into standardized Skills. This atomic decoupling capability perfectly matches the multi-strategy, high-frequency iteration nature of the recommendation business.
Enterprise-Grade 'Deterministic' Governance: Flexibility is the sharp edge of an Agent, but in a production environment, determinism is the cornerstone. The OpenClaw framework provides a more rigorous control mechanism for the ReAct logic flow (e.g., maximum iteration circuit breaker, tool call whitelist), effectively preventing the Agent from falling into logical infinite loops or generating uncontrolled hallucinations.
Ecological Compatibility and Capability Reuse: OpenClaw's adaptability to evolving standards ensures that the Skill assets we accumulate can not only serve 'TuiChaCha' but also radiate to other internal intelligent scenarios at low cost in the future, achieving 'one set of core capabilities, called upon across the entire domain.'

3. Highway and ATV, the 'Fast' and 'Thorough' Dual Modes of Diagnosis

In system design, developers often face a choice: pursue a linear logic for ultimate response speed, or pursue autonomous intelligence for deep reasoning? Our answer is: refuse to compromise, embrace both. Based on this concept, 'TuiChaCha' has built a unique 'Highway & ATV' hybrid agent architecture. It is not a simple stacking of two modes, but a deep integration of efficiency and intelligence.

Highway: This is our standardized diagnostic pipeline, specifically designed to handle 80% of common problems. When a user's question matches a standard Story in the knowledge base (e.g., 'evaluate product based on screenshot,' 'inaccurate recommendation for a user,' 'a product was filtered'), the system automatically enters Highway mode. It executes step-by-step according to a preset standard script (Story), like a factory assembly line.
ATV (All-Terrain Vehicle): This is our ace scout, specifically tackling the 20% of long-tail, complex hard bones. When encountering an unprecedented anomaly, such as the bizarre scenario of 'feature service network jitter causing a collective drift in fine-ranking scores,' Highway's fixed script fails. At this point, the system automatically and seamlessly downgrades to ATV mode. Based on the ReAct framework, it plans autonomously like a true expert: first check logs, discover network errors; then call tools to check the network topology; then correlate and analyze other failed requests in the same period... reasoning step by step until the root cause is found. It does not rely on fixed scripts, being flexible and powerful.

The key lies in the intermediate intelligent scheduler (judgment engine). When a user's question comes in, it first quickly matches against the knowledge base. If a standard Story is hit, it takes the Highway; if no match is found or the Highway execution gets stuck, it immediately switches to ATV mode. Dual-track complementarity and intelligent switching ensure that any problem has a path to follow and a method to use.

'TuiChaCha' Highway & ATV Dual-Track Architecture Diagram

The left side's highway handles common problems, the right side's off-road vehicle tackles complex anomalies, and the middle intelligent scheduler achieves seamless switching.

4. Core Technology Dissection: A Hardcore Breakdown from Principle to Implementation

Having told the story of the 'AI diagnostic expert,' it's time to open the hood and see how this 'diagnostic sports car' actually runs.

Story Foundation: Atomic Skills

We have built a set of Skill tools based on internal APIs that follow the MCP standard (due to data security issues, they were not uploaded to the company's Skill library). This tool library acts like a 'universal interface' for recommendation capabilities, not only supporting the precise scheduling of the current Agent but also forming a reusable skill base. Through simple field adaptation, this capability can rapidly radiate to scenarios like communities and advertising (welcome to communicate), becoming a 'booster' for various fields to quickly launch their own Agent capabilities. Several core Skills are introduced below:

These Skills are like Lego bricks; Highway's scripts can be assembled in order, and ATV's reasoning engine can call them on demand. Need a new troubleshooting capability? Just encapsulate another Skill and add it in—hot-swappable, no downtime.

Highway Story Orchestration: Script-Driven Deterministic Execution

The core of the Highway mode is the Story mechanism — solidifying an expert's troubleshooting path into a reusable standard script. Each Story is essentially an ordered Skill orchestration template, defining a standard execution path.

Story Structure Example:

Story: Product Recommendation Anomaly - Standard Diagnosis ├── Step 1: screenshot-trace-query # Screenshot recognition, auto-match Trace ├── Step 2: user-profile-query # Check user purchase/click history ├── Step 3: layer-items-query # Check product exposure at each layer ├── Step 4: strategy-logs-query # Verify strategy intervention (dedup/filter) ├── Step 5: commodity-forward-query # Check product forward index info └── Step 6: output_diagnosis # Output diagnostic conclusion

Core Features of Story:

Deterministic Execution: Each Step is executed strictly in the orchestrated order; there is no 'AI autonomous decision-making.'
Conditional Branching: Supports if/else conditional judgments, e.g., 'If recall result is empty → enter recall-specific troubleshooting branch.'
Hot-Swappable: Stories can be independently version-controlled. When business iterates, only the Story configuration needs updating, without code changes.
Fallback: When a Step execution fails or confidence falls below a threshold, an ATV downgrade is automatically triggered.

How Stories are Born:

Manual Accumulation: Experts (i.e., our R&D colleagues) manually orchestrate core troubleshooting paths.
Automatic Evolution: After ATV successfully resolves a long-tail problem, a new Story is automatically distilled and generated through a reflection framework.

This is the essence of 'Highway' — turning the 'experience' in an expert's mind into an executable, reusable, and evolvable standardized script.

ATV's ReAct Loop: Giving AI 'Rules to Follow'

The ATV mode uses the OpenClaw framework to implement the classic ReAct loop:

Question → Thought → Action(Skill) → Observation → Final Answer

But an 'unbridled' AI easily falls into an infinite loop, like getting lost in a maze and obsessively exploring every fork, forgetting to find the exit. We equipped it with three constraints.

Max Iterations = 6: A maximum of 6 thought steps; even the strongest expert must stop to prevent 'over-reasoning.'
Tool Whitelist: Only give it the small tools it can use, preventing random calls (e.g., preventing access to irrelevant data).
Confidence Threshold: Each reasoning step must self-question 'Is this conclusion reliable?' If not, change the train of thought, refusing blind confidence. This mechanism allows ATV to explore autonomously without running wild. After all, we want an 'off-road vehicle,' not a 'runaway horse.'

Auxiliary Diagnosis: Vector Retrieval in the Knowledge Base

As TuiChaCha deepened its involvement in the business, the accumulation of technical documents, code architecture, and business memory became a bottleneck. For example, the recall channel TIDE, it didn't know it was the tidal recall for the homepage waterfall flow, and would only interpret it literally as a non-personalized recall based on site-wide popularity/trends; simultaneously, traditional RAG solutions had obvious shortcomings in code understanding and cross-scenario association. We explored a hybrid architecture of 'Context Engine + Knowledge Graph.'

Core Components

OpenViking is not a traditional RAG tool, but a local context engine centered on 'context-driven,' integrating memory storage, document indexing, semantic retrieval, and multi-Agent isolation and sharing. It mainly serves AI application scenarios requiring long-term memory, large-scale document retrieval, and multi-Agent collaboration, and is particularly adapted to the OpenClaw plugin ecosystem, solving problems like weak native memory retrieval and difficulty in multi-Agent data isolation.

Graphify (often called Graphify Viki by the community) is an open-source tool that exploded in popularity in 2026. Its core positioning is to transform local folders (code, documents, images, etc.) into a knowledge graph of 'entities + relationships,' focusing on extreme token saving, code engineering understanding, and multi-modal adaptation, precisely compensating for OpenViking's shortcomings in deep parsing of technical documents.

We used the combination of OpenViking + Graphify to reconstruct the construction logic of the local knowledge base — OpenViking is responsible for 'managing memory, managing collaboration, managing retrieval,' while Graphify is responsible for 'doing structuring, doing parsing, doing simplification.' The synergy of the two achieves:

Solving the pain point of 'memory extraction failure' in technical documents: Graphify performs structured decomposition, and OpenViking stores and retrieves, enabling effective use of large technical content.
Extreme Token Saving: Graphify's simplified graph + OpenViking's three-tier hierarchy significantly reduces LLM context consumption and improves response speed.
Efficient Multi-Agent Collaboration: OpenViking implements data isolation and sharing, while Graphify provides a unified structured knowledge base, adapting to complex business scenarios.

The entire knowledge base content is divided into three layers by detail level, loaded on demand to avoid context bloat.

In the past, troubleshooting was a two-step process: manually locate the problem, then independently look up the solution. Now, AI completes root cause identification and repair guidance in one stop, simultaneously providing the anomaly location and the corresponding execution plan. For example: it won't just coldly tell you: 'Conclusion: Blocked by the deduplication strategy.' It will combine the company's business Wiki and SOP manual to add: 'Action Suggestion: Please go to the product management platform to confirm and check if the product ID was mistakenly added to the 'Product Search/Recommendation Blacklist'.' It has paved the last mile from diagnosis to repair for you. Now, for many simple problems, product and operations colleagues can figure them out themselves, and we are finally freed from the work of 'repetitive Q&A.'

Subsequent Optimization Points: The way we currently use the knowledge base is semantic triggering. When asking TuiChaCha a question, as long as the question involves memory retrieval or background knowledge queries, OpenViking is automatically triggered. This method has a triggering strategy problem; TuiChaCha 'doesn't know when to check documents' or 'checks when it shouldn't, and doesn't check when it should.' There are a few ideas now, to be explored and tested for effectiveness later, and everyone is welcome to discuss:

Introduce 'Confirmatory Retrieval': 'Before answering a technical question, you must first assess: 'Do I have the latest configuration parameters/documents for this question?' If the answer is NO, you must call the knowledge retrieval Skill; blindly answering from memory is prohibited.' Core logic: Utilize the Agent's Chain of Thought (CoT) to add a 'self-questioning' step before checking documents; Standard instruction: Add a fixed constraint in the System Prompt; Effect: Force the Agent to recognize its own 'knowledge blind spots,' changing passive retrieval to active requesting.
Intent Classification Triggering: Don't let the Agent scan the vector database for every sentence; instead, pass it through a lightweight intent judgment layer first; Explicit Triggering: Train a micro-classifier (or constrain via System Prompt) to recognize instructions with characteristics like 'query,' 'confirm,' 'compare,' 'background'; Keyword Density: If the user input contains domain-specific proper nouns or long-tail words (high IDF words), force trigger retrieval; Disadvantage: Cannot handle implicit background needs.

Evolution Layer: The Full Evolution Process from 'Troubleshooting Records' to 'General Methodology'

The core of the evolution layer is the three-stage closed loop of 'Reflection - Abstraction - Verification,' referencing Hermes' self-evolution mechanism, allowing the system to learn 'general methodology' from 'troubleshooting records':

Step 1: Trajectory Pruning and Efficiency Purification

Core Task: Compress the lengthy Agent thought process into a 'golden path.'
Action: The evolution engine backtracks through all the Agent's troubleshooting history, satisfaction feedback, and daily memories from the original investigation.
Logic: Identify and eliminate trial-and-error paths (e.g., querying irrelevant product information), retaining only the tool call chain that played a decisive role in locating the final root cause.
Effect: Compress a cumbersome process that originally required 10 steps into an efficient 3-4 step diagnostic chain, achieving 'optimization of the troubleshooting path.'

Step 2: Feature Dimension Elevation and Logic Generalization

Core Task: Break away from the contingency of a single case and extract universal business logic.
Action: Utilize the LLM's business perception capability to automatically map specific 'hard-coded data' (like a specific UID, a specific RequestID, a specific product ID) to high-dimensional business variables.
Significance: Solve the 'spatial transferability' of logic, ensuring that this troubleshooting experience can automatically adapt to similar failures for different users and different time periods.

Step 3: Story and Skill Generation

Action: Encapsulate the generalized logic chain into a Story and Skill conforming to standard protocols.
Logic: Instead of writing complex code, generate a structured skill.md containing 'trigger conditions, execution logic, assertion checks.'
Output: skill.md, which can be directly stored in the Agent's Skill Store for sub-second invocation in subsequent troubleshooting.

Step 4: Sandbox Dry Run and Admission Verification

Core Task: Establish a quality defense line for knowledge to prevent immature diagnostic logic from polluting the system.
Action: Before a new skill officially goes live, the system automatically captures relevant troubleshooting cases from the past 7 days.
Replay: Force the system to use the newly generated Story or Skill to automatically replay historical cases.
Admission Criteria: Is the consistency between the automated troubleshooting conclusion and the expert's manual conclusion > 90%? Decision: If the standard is met, merge into Highway (high-speed diagnostic channel); if not, return to the evolution layer for re-reflection.

From then on, every time the system solves a new problem, it automatically learns a new skill and automatically optimizes existing skills. Knowledge no longer gets lost with personnel changes but is systematically accumulated within the system. It truly achieves 'the more it's used, the smarter it gets,' evolving from a 'tool' into an intelligent agent with 'diagnostic capability.'

Why Didn't We Directly Copy Hermes?

Actually, before Hermes came out, we already had a preliminary daily summary and reflection mechanism, but Hermes still gave us great inspiration, especially its 'reflection-distillation' mechanism. However, in the production environment of a recommendation system, what we need is 'deterministic evolution.'

Hermes is like 'practicing internal strength': It makes the model's overall quality stronger, but you don't know what move it will use in the next troubleshooting.
OpenClaw is like a 'blacksmith's shop': It gave us the best tool molds.
TuiChaCha is like 'writing a boxing manual': We use the ReAct framework provided by OpenClaw to let ATV 'practice in real combat,' then refer to Hermes' ideas to prune and generalize those effective moves (paths) from real combat, finally solidifying them into sections of Highway's 'boxing manual' (Story).

Our evolution layer has its own 'TuiChaCha characteristics':

Detachment from Parameter Dependence: The results of our evolution are Skills and Stories. This means even if we change the underlying large model tomorrow, these accumulated 'troubleshooting experiences' can still be seamlessly migrated.
Replay Admission System: Unlike Hermes' fully automatic merging, we added a 'Dry Run' verification step. A newly evolved skill must maintain over 90% consistency in replaying past real cases before it can officially enter the Highway channel.

Senior Engineer's Note: 'True evolution is not blindly pursuing automation, but achieving the streamlined accumulation of knowledge while maintaining system determinism.'

5. Real-World Case: Diagnosis of Duplicate Product Recommendations

This is a real case from the internal testing group: 'Why does it keep pushing phone cases to me?' TuiChaCha automatically executes the diagnostic process.

Diagnostic Conclusion (AI Auto-Generated):

The phone case was recalled by the TIDE channel (TIDE = Tidal Recall, a secondarily processed recall channel);
The original recall source is stored in the item.extMap.cns_v2 field;
The system found that the user had recently browsed phone case products (interest signal) and continuously recalled similar products;
Conclusion: Not a bug, but a normal recall triggered by user behavior signals, just that the exposure frequency was too high, affecting user experience.

6. Final Words: When AI Becomes the System's 'Diagnostic Expert'

Looking back at the growth path of 'TuiChaCha,' it is not just a troubleshooting tool. It is an intelligent diagnostic system with diagnostic capability, learning ability, and even some 'practical experience.' It frees us engineers from the repetitive, inefficient, and uncertain 'metaphysical debugging,' allowing us to focus more on the iteration and innovation of the algorithms themselves. Of course, 'TuiChaCha' is still continuously growing and iterating, and occasionally there are still some unexpected answers. Its significance lies in the fact that we are starting to use AI to solve the most troublesome troubleshooting problems of the recommendation system itself. This might be a feasible path towards AI for System, and even autonomous operation and maintenance in the future AGI era. The next step, we can expect it to move from 'diagnosis' to 'prevention,' issuing warnings before problems occur.

Isn't the essence of technological development to free people from tedious, repetitive labor so they can engage in more creative work? 'TuiChaCha' has brought us one small step closer to this goal.

Past Reviews

Text / Yan Yang

Follow Dewu Technology for weekly technical干货 (practical content)

If you find the article helpful, feel free to comment, forward, and like ~

Reproduction without permission from Dewu Technology is strictly prohibited, otherwise legal liability will be pursued according to law.