Artificial Intelligence

Intent Routing at Scale: The Production Architecture That Replaces a Naive Prompt

By 神奇小汤圆 · Jul 22, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Agent systems that route user requests to dozens or hundreds of backend skills are becoming the default architecture for enterprise copilots. The naive prompt approach burns budget and silently routes errors; the layered routing pattern described here is the difference between a hackathon demo and a system that survives contact with real users.

Summary

The demo approach to agent intent recognition — stuffing all intent descriptions into a single prompt — collapses under real-world loads. Token costs explode linearly with intent count, accuracy degrades as similar intents blur together, and a single misclassification gets routed straight into the wrong agent with no safety net.

A production-grade design replaces that flat prompt with a three-layer pipeline. Deterministic rules catch obvious requests for free. Embedding-based semantic recall narrows hundreds of intents to a top-5 candidate set, slashing the prompt size the LLM must process. The LLM then fine-ranks only those candidates and returns a confidence score. When confidence is low, the system does not guess — it asks the user to clarify, preventing silent misroutes.

Beyond the routing logic itself, a maintainable system needs full traceability of every decision, an observability dashboard tracking latency, token burn, and clarify rates, an offline evaluation loop built from sampled production traces, and a golden benchmark dataset that gates every prompt or model change with a regression test.

Takeaways

— Putting all intents into one prompt causes token cost to scale linearly with intent count — 100 intents at 300 tokens each burns 30,000 tokens before the model even sees the user query.

— More intents in a single prompt lowers classification accuracy because semantically similar intents (e.g., weather, forecast, air quality) bleed into each other.

— Intent descriptions must be mutually exclusive with minimal overlap; fuzzy boundaries are the top cause of misclassification in production.

— A rule-matching layer (regex, keywords, allowlists/blocklists) catches deterministic requests instantly without touching an LLM, saving cost and latency.

— Embedding-based semantic recall reduces a candidate set of hundreds of intents to a top-5 or top-10 list before the LLM ever sees it.

— The LLM fine-ranks only the recalled candidates and outputs three fields: the chosen intent, a confidence score, and a reason for the decision.

— Low-confidence results (e.g., 55%) trigger a clarify step that asks the user to disambiguate rather than silently routing to the wrong agent.

— Every routing decision should be traced end-to-end: user input, embedding top-K, the full LLM prompt, the LLM output, the agent entered, tools called, and final reply.

— An observability dashboard tracks QPS, P95 latency, embedding and LLM call times, token consumption, timeouts, retries, fallbacks, and clarify rate to catch anomalies early.

— Offline evaluation uses sampled production traces with human annotation or LLM-as-a-Judge to compute accuracy, precision, recall, and F1 on a ground-truth dataset.

— A golden benchmark dataset covering normal queries, ambiguous queries, typos, mixed-language input, and unknown intents gates every prompt or model change with a regression test.

— The full system is an engineering closed loop: production traces feed offline evaluation, which enriches the benchmark, which gates deployments.

Conclusions

Most agent intent-recognition advice stops at the prompt, but the real engineering is in the routing architecture that sits in front of the LLM — rule matching and embedding recall do the heavy lifting so the model only sees a handful of candidates.

The clarify step is under-discussed. Refusing to guess on low-confidence classifications and instead asking the user turns a silent failure mode into a recoverable interaction, which is essential for any agent that triggers side effects.

Observability for LLM pipelines is still immature. The metrics proposed here — clarify rate, token burn per request, embedding vs. LLM latency split — are a practical starting point that most agent frameworks do not yet expose natively.

Treating intent routing as a continuous-evaluation problem rather than a one-time prompt-engineering task is the dividing line between a prototype and a system that can be safely iterated on by a team.

Concepts & terms

Multi-stage Intent Routing

A pipeline that narrows the intent classification problem across successive layers — typically rule matching, embedding-based semantic recall, and LLM fine-ranking — so the LLM only compares a small candidate set instead of every possible intent.

Embedding Semantic Recall

Using vector embeddings to compute the similarity between a user query and all intent descriptions, then returning only the top-K most similar intents as candidates for a downstream LLM to classify.

Clarify (Disambiguation Fallback)

When an intent classifier's confidence is below a threshold, the system asks the user to disambiguate (e.g., 'Did you mean Apple the company or apple the fruit?') instead of silently routing to a potentially wrong agent.

Golden Dataset / Benchmark

A curated set of labeled queries — including normal, ambiguous, misspelled, and out-of-scope examples — used to regression-test an intent routing system before any prompt, embedding model, or LLM change is deployed.

LLM-as-a-Judge

Using a large language model to evaluate or label outputs (e.g., classifying whether a routing decision was correct) as a faster, cheaper complement to human review when building evaluation datasets.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗