From Rigid Scripts to Service Partners: Deploying a Controllable Multi-Agent Customer Service System
Most agent demos collapse under real customer-service loads: long contexts, multi-turn negotiations, and the need to sound human while staying on-policy. This stack shows a repeatable path from a naive LLM call to a trained, evaluated, and controllable multi-agent system, with concrete numbers on resolution rates and alignment with top human performers.
Traditional customer-service bots break on multi-step tasks and sound robotic. Dewu's team replaced a single-prompt Qwen agent with a multi-agent architecture built on AutoGen, using a master controller, response, evaluator, and polishing agent to handle complex negotiations. The system now runs under a cross-scenario Harness that adds cross-session memory, context compression, and a data flywheel for continuous improvement.
A prompt-engineering automation pipeline compares bad agent replies against good human replies, proposes prompt changes, and validates them offline before online experiments. On the training side, DPO and GRPO-based RL teach the model correct decisions, with specific countermeasures against reward hacking and catastrophic forgetting. A separate distillation pipeline aligns the agent's tone, emoji use, and decision logic with the top 5% of human agents.
Half-duplex message control handles the reality that users can fire off multiple messages while the agent is still generating a reply. A dedicated Qwen3-4B classifier decides when to merge or split user messages, keeping the conversation coherent without dropping context.
Prompt engineering, often treated as a craft, is reframed here as a data-flywheel problem: the system proposes prompt edits from outcome diffs, and humans only review, which turns prompt tuning from a bottleneck into a continuous loop.
Training separate discriminators for reply-ability, correctness, and human-likeness, then using them to construct DPO pairs, is a practical decomposition that avoids trying to capture all quality dimensions in a single reward signal.
The half-duplex design acknowledges a real UX constraint most chatbot frameworks ignore: the agent's inference latency means user messages pile up, and a dedicated classifier for merging or splitting them is cheaper and more reliable than trying to handle it inside the main agent prompt.