1. Current Pain Points of Intelligent Customer Service

Can't understand human speech, acts too clever: Unable to accurately understand user intent, frequently gives irrelevant answers. High labor costs, severe queuing: Human resources cannot cover high-concurrency scenarios. Weak problem-solving ability: Can only handle simple FAQs, powerless against complex multi-turn problems.

2. The Inevitability and Challenges of Implementing an Agent for Intelligent Customer Service

The Inevitability of Agent-based Intelligent Customer Service

Traditional architectures have an upper limit: the pipeline architecture of intent classification + NER + dialogue management + FAQ recall + re-ranking cannot handle multi-step tasks, and context connection is poor.

SOP manual maintenance costs are high: scenario-based queries cannot be exhaustively listed. Multi-model operation and maintenance costs are high: multiple small models are deployed separately, "a slight move in one part may affect the whole situation," making O&M extremely costly. Conversation human-likeness/fluency is poor: responses are monotonous and cannot replicate the decision-making and communication skills of excellent human agents.

Challenges of Agent-based Intelligent Customer Service

High cost of optimizing instructions for complex scenarios: scenarios are complex, knowledge processes are cumbersome, and the PE optimization cost for long-tail cases is high. Insufficient long-context multi-turn capability: multi-turn decision-making/negotiation combining diverse user needs and long historical contexts is a major challenge. Insufficient ability to align with human-like communication: human responses are highly diverse, with flexible use of nicknames, scripts, emojis, and modalities, requiring a "human-assisted Agent." Message control under multi-turn half-duplex: inference time is long, and in a half-duplex environment, controlling message interruption/context topic switching is difficult.

3. Multi-turn Negotiation Optimization for Customer Service Agent (Highly Controllable PE Automation)

Single-Scenario Architecture Evolution

Phase 1: Single-Agent (Single Prompt)

A Single-Agent architecture based on Qwen, with problems including:

• Prompt is too long, and the model's instruction-following ability is limited.
• Serial Pipeline cannot implement Agent multi-turn ReAct.

Phase 2: Multi-Agent (Progressive Disclosure)

Adopted AutoGen's Multi-Agent architecture, introducing:

• Master Control Agent: Responsible for task scheduling and process control.
• Response Agent: Responsible for generating the final reply script.
• Evaluator Agent: Performs real-time evaluation of output quality.
• Polishing Agent: Enhances the naturalness and human-likeness of replies.

Dynamic scheduling is achieved through the SelectorGroupChat mode, significantly improving the resolution rate.

Cross-Scenario Harness Architecture

To solve the problem of cross-scenario reusability, the architecture further evolved into the Harness architecture, with core capabilities including:

• Cross-session Memory: Supports user historical context memory.
• Context Compression and Trajectory Compression: Effectively manages long conversations.
• Master Control Agent Management: Improves reply controllability.
• Data Flywheel drives continuous model optimization.
• Skill Automated Generation (in progress).

Data Flywheel: PE Automation and Model Training

PE Automation Optimization

Manual Prompt Engineering has obvious pain points: high time cost. Multiple cases cause Prompt conflicts and ambiguity. Token cost and instruction-following ability are limited.

To this end, we built a PE automation pipeline. Comparative data extraction: LLM extracts Agent replies (Bad) and human replies (Good) from cases transferred to human agents. PE modification suggestions: LLM analyzes Prompt modification suggestions. Manual Review: Humans annotate and check feasibility. Offline run tests: Merge modification suggestions and run tests on the same batch of data. Online experiments: Accumulate data and continuously optimize.

Effect Verification:

Data Flywheel - Model Training

Through the DPO model training flywheel, "LLM-as-a-Judge" is used to collect feedback data and retrain:

• Can-reply discrimination model.
• Reply correctness model.
• Human-likeness discrimination model.
• Construct chosen and reject samples.
• Train polishing model.

4. RL Decision Training: Teaching the Agent to "Make Correct Decisions"

Training Framework

The RL training framework is divided into three major modules:

Data Infrastructure

• Online feedback data collection, strategy instrumentation to construct Multi-turn RL training data.
• Expert model generates decision reasons for CoT-RL training.
• MCP Server deploys real-time Tools calls.

Model Training

• Cold-start SFT warm-up.
• Verifiable Reward: Rule-Based-Reward.
• Generative Reward: Model-Based-Reward.
• GRPO training.

Evaluation System

• LLM-as-a-Judge.
• Human evaluation.
• User simulator consistency evaluation.

Countering Reward Hacking

• Optimize Reward.
• Solve data bias problems.
• Add thinking process - CoT training.

Training Effect Comparison

Experiments

Ablation Experiments

• Comparison of different epochs.
• Comparison of different cold-start SFT base models.
• Comparison of different learning rates.

Business Interference Ablation Experiment

• Business A.
• Business B.
• Business A + Business B.

RL Training Reduces Catastrophic Forgetting

Test Standards: C-Eval, CMMLU.

Mitigation Methods:

• Reduce learning rate (full-parameter training ≤ 5e-5).
• Monitor general benchmark metrics in real-time via eval-step.
• Limit training to 2~4 epochs.
• Data ratio control: Model self-reasoning VS Model relying on Tool calls = 1:1.

5. Emotional Temperature: Aligning Human Experience through Model Distillation

Service Evaluation Standards

A complete 100-point evaluation system was established:

Model Distillation Training

Three training modes work collaboratively: CoT-RL Strategy Distillation: Distills the decision-making logic of excellent customer service agents; DPO Script Distillation: Aligns conversation tone, reduces repetition, and standardizes the use of emojis and titles; Emoji Module: Emoji classification + context analysis, finely controls the timing of emoji usage.

Emoji Classification System: Opening welcome / Positive and happy / Thinking and puzzled / Apologetic and soothing / Reminder and emphasis / Expectant and waiting / Completion confirmation / Working hard / Logistics delivery / Price discount / After-sales handover / Ending thanks.

Human Alignment Effect

Model scores are close to the level of the Top 5% of excellent human agents.

6. Multi-turn Message Flow Control: Half-Duplex Dialogue Logic Design

Message Sending and Receiving Mode

Specificity of the intelligent customer service scenario: Users can send messages at any time (full-duplex), but the Agent should not be interrupted during a reply (half-duplex); User→Agent: Full-duplex, can interrupt; Agent→User: Half-duplex, cannot be interrupted.

Multi-Message Processing Strategy

To achieve accurate message merging/splitting judgment, a dedicated classification model is trained:

• Data Collection: Pull online user consecutive messages for annotation.
• Data Augmentation: LLM expansion of negative samples.
• Model Training: Qwen3-4B classification training.

7. Summary and Outlook

Summary of This Practice

This sharing revolves around the following core technologies: Customer Service Agent challenges and architecture design (Single-Agent→Multi-Agent→Harness); Highly controllable data flywheel (PE automation + DPO training); RL strategy training and human experience alignment; Multi-turn half-duplex message flow control.

Future Outlook