跪拜 Guibai
← Back to the summary

Intent Routing at Scale: The Production Architecture That Replaces a Naive Prompt

Interviewer: How do you handle intent recognition for an Agent?

If it were you.

The interviewer suddenly asks:

"How do you handle intent recognition for an Agent?"

Is your first reaction something like this?

"Just put all Intents and Intent Descriptions into the Prompt and have the large model return an Intent."

You finish speaking.

You think you've answered.

The interviewer nods.

Then continues asking:

"What if there are dozens or hundreds of Agents online?" "What if there are a lot of Intents?" "What if two Intents are very similar?" "What if the recognition is wrong?"

......

If you can't answer any of these questions.

Basically, this round of the Agent interview is over.

Many people think they just don't know the answer.

Actually, that's not it.

It's that your answer stays at the Demo stage, while the interviewer wants to hear a production-grade solution.

Today, let's review how this question should actually be answered.

The First-Level Answer (95% of people answer this way)

Many people will say:

Intent recognition for an Agent is actually very simple.

For example, there are several Intents in the system.

Query Weather
Query Stocks
Chat
Knowledge Q&A
Generate Code

Then write a description for each Intent.

For example:

Intent1:
Query Weather

Description:
The user wants to query weather, temperature, air quality.

Intent2:
Stock Query

Description:
The user wants to query stocks, funds, securities information.

Then put all Intents together into the Prompt.

For example:

System Prompt

You are now responsible for intent recognition.

Below are all Intents.

......

User Input:

What's the weather like in Shanghai today?

Please return the Intent name.

The LLM returns:

Weather

Then the system enters the Weather Agent.

The whole process ends.

Many Demos are done this way.

In fact, for an Agent with only a few Intents, this solution is completely fine.

Many open-source projects are also implemented this way.

But.

If you stop your interview answer here.

It's basically over.

Because in real online systems, almost no one does this.

What does the interviewer really want to hear?

A real Agent doesn't just have a few Intents.

It has dozens.

Even hundreds.

For example, an enterprise Copilot.

It might have:

HR Agent

Finance Agent

Contract Agent

OA Agent

Approval Agent

CRM Agent

R&D Agent

Operations Agent

......

Each Agent has dozens of Intents.

If you put them all into the Prompt.

Several problems immediately arise.

First Problem: Token Explosion

Assume:

One Intent description is 300 Tokens.

100 Intents.

That is:

300 × 100

=30,000 Tokens

The user only asks one sentence:

What's the weather like in Shanghai today?

But you first send thirty thousand Tokens.

The cost explodes.

Not only is it expensive.

It's also slow.

Many engineering teams will first do routing, sending the request to a smaller capability set, rather than handing all capabilities to the LLM at once.

So.

The first thing.

You must reduce the number of Intents participating in classification.

Second Problem: More Intents, Lower Accuracy

For example:

Query Weather

Weather Forecast

Air Quality

Life Index

Future Weather

The LLM can easily get confused.

Especially when Intent boundaries are unclear.

Accuracy will get lower and lower.

Therefore, production environments generally require:

Intents must be mutually exclusive.

Descriptions must be clear.

There cannot be a lot of overlap.

This is the most emphasized point when many enterprises do Intent Design.

Third Problem: What to do after misrecognition?

Many people don't know what to do here.

In a real online system.

You don't trust a single classification.

Because:

An LLM can always judge incorrectly.

So there must be:

Confidence

For example:

Intent:

Weather

Confidence:

98%

If:

98%

Directly enter the Agent.

If:

55%

What to do?

Don't continue executing.

Instead:

Ask the user back.

For example:

Are you looking for today's weather,

or the weather for the next seven days?

Or:

The 'Apple' you mentioned,

is it Apple Inc.,

or the fruit?

Don't guess.

Let the user confirm.

This is a very important step for online Agents.

The answer that truly satisfies me is this layered architecture

If I were the interviewee.

I would answer like this.

The simplest solution is indeed to hand all Intents and Intent Descriptions to the LLM for classification. But this approach is more suitable for a Demo. If it's a production environment, I generally adopt Multi-stage Intent Routing.

The whole process is as follows:

First, the user inputs a sentence, for example, here it is:

"How much is Apple?"

Note that this sentence is actually ambiguous.

It could be asking about the price of the fruit Apple, or it could be asking about the stock price of Apple Inc., or even the price of an iPhone.

So, if you throw all Intents at the large model for judgment right from the start, it's not only costly but also prone to misjudgment.

Therefore, production environments generally don't do this.

The first layer is called Rule Matching.

The only goal of this layer is: Quickly filter deterministic requests.

Like Regex, keyword matching, allowlists/blocklists, etc.

If a sentence can directly determine the intent through rules, for example, fixed expressions like "weather forecast" or "help center", then the system directly returns the result or enters the corresponding Agent, without needing to call the large model at all.

The biggest benefit of this is high speed and low cost.

If the rules don't match, then it goes to the next layer.

The second layer is Embedding Semantic Recall.

This layer is very critical.

Suppose your system has 100 Intents. If you hand them all to the LLM for comparison, not only is the Prompt very long, but the Token consumption is also very high.

So here, Embedding is first used to calculate the semantic similarity between the user input and all Intents.

For example, from 100 Intents, only the most relevant Top 5 are recalled.

This layer doesn't make the final decision, but is responsible for narrowing the search scope.

You can think of it as the "initial screening" in a search engine, filtering out all obviously irrelevant Intents.

The third layer is where it's truly handed to the LLM for classification.

But at this point, the large model no longer has to face 100 Intents, but only needs to choose among the 5 candidate Intents just recalled.

Therefore, it can output three results:

The first is the finally recognized Intent;

The second is the Confidence;

The third is why this judgment was made, which is the Reason.

This way, not only is the accuracy higher, but the Prompt is shorter, and the inference speed is faster.

Finally, the system handles things differently based on Confidence.

If the confidence is high, for example, over 80%, then it directly enters the corresponding Agent to execute business logic.

But, if the confidence is low, never guess.

Instead, enter the Clarify stage, actively confirming the real intent with the user.

Like the example here:

"The 'Apple' you mentioned, is it Apple Inc., or the fruit?"

After the user answers, perform intent recognition again.

Although it adds one more round of dialogue, it can greatly reduce the business risk caused by misjudgment. This is a design adopted by many enterprise-level Agents.

So, the core idea of this diagram is actually one sentence:

Don't let the LLM judge all Intents from the start, but achieve the best balance between accuracy, Token cost, and response speed through a multi-stage routing architecture of "Rule Filtering + Embedding Recall + LLM Fine-Ranking + Clarify Fallback".

This is also why what many people build is a Demo, while Agents that truly go online almost always adopt this layered Intent Routing architecture.

Why add Embedding?

Many people will ask:

Since the LLM does the classification in the end anyway.

Why add Embedding?

The answer is just two words:

Cost reduction.

For example:

Online:

500 Intents

Embedding recalls once:

Top10

The LLM only needs to compare:

10

The Prompt is greatly reduced.

Tokens are greatly reduced.

Faster speed.

Lower cost.

Many production systems adopt a two-stage or three-stage routing like "Embedding Initial Screening + LLM Fine-Ranking" or "Lightweight Classifier + LLM Fallback", rather than letting the LLM face the complete intent set every time.

If the interviewer continues asking: "Can it be further optimized?"

If the answer reaches here, many candidates already don't know what else to say.

But a true enterprise-level Agent is far from finished just by recognizing the Intent.

In a production environment, an excellent Intent Routing system must also possess the capabilities for continuous observation, continuous evaluation, and continuous iteration.

If it were me, I would continue answering like this.

① Establish complete Traces, able to fully replay every decision process

First, I would establish a complete Trace for every request.

What's recorded is not just the finally recognized Intent, but the entire Routing decision chain.

For example, one request would record:

User Input
        │
        ▼
Embedding Top-K Recall Results
        │
        ▼
LLM Prompt
        │
        ▼
LLM Output
(Intent, Confidence, Reason)
        │
        ▼
Which Agent entered
        │
        ▼
Which Tools called
        │
        ▼
Final reply result

The greatest value of doing this is Traceability.

For example, one day a user reports:

"Why does this Agent always misunderstand my question?"

At this point, we can directly replay this Trace.

Did the Embedding not recall the correct Intent?

Or did the Prompt cause the LLM to classify incorrectly?

Or was the Confidence threshold set too low?

With a complete Trace, the entire decision process can be restored, and the efficiency of troubleshooting will be much higher. This is also why many Agent platforms first build Tracing capabilities, because subsequent evaluation, debugging, and optimization all depend on complete run trajectories.

② Establish an online Observability system

When many people mention model deployment, they immediately think of Accuracy.

But actually, online systems are more concerned with:

Is the system running normally today.

Therefore, we usually build an Observability Dashboard to continuously monitor the running status of the entire Intent Routing.

For example:

These metrics can be collected directly from the system's running process without manual intervention.

For example, when the Clarify Rate suddenly rises sharply in a short period, although it doesn't directly indicate a model classification error, it at least shows that the number of requests entering the clarification process has significantly increased recently.

At this point, we can further analyze:

Was the Prompt recently modified?

Was a batch of easily confused Intents newly added?

Or after a model upgrade, did the distribution of Confidence change?

Therefore, the role of online Observability is not to directly judge whether the model is right or wrong, but to detect anomalies early and help R&D quickly locate problems. Current mainstream Agent platforms also emphasize that the online stage should mainly consist of Traces, system running metrics, and reference-free evaluations.

③ Establish an offline Evaluation system

Online is responsible for discovering problems.

Truly judging the model's capability is done offline.

Usually, we periodically sample some real user requests from online Traces to build an Evaluation Dataset.

The whole process is as follows:

Online Production Trace
        │
        ▼
Sample real Queries
        │
        ▼
Human annotation
(or LLM-as-a-Judge initial screening)
        │
        ▼
Generate Ground Truth
        │
        ▼
Re-run the latest Router
        │
        ▼
Calculate Accuracy, Precision, Recall, F1

With Ground Truth, we can objectively evaluate the current model's recognition capability.

At the same time, we can also discover which Intents are most easily confused and which Queries are most prone to misjudgment, providing a basis for the next round of optimization.

Currently, many teams adopt a combination of Human Review + LLM-as-a-Judge to improve evaluation efficiency while ensuring evaluation quality.

④ Establish your own Benchmark to ensure no degradation with each upgrade

Finally, I would also maintain a long-term evolving Benchmark.

For example, maintain a Golden Dataset, which collects a large number of real business scenarios, including:

Every time you modify the Prompt, adjust Intents, upgrade the Embedding model, or switch to a new LLM, you must re-run the complete set of Benchmarks.

Only when the accuracy hasn't dropped, performance hasn't significantly degraded, and costs still meet expectations, is the new version allowed to go online.

This forms a complete continuous optimization closed loop:

Production Trace
        │
        ▼
Offline Evaluation
        │
        ▼
Golden Dataset
        │
        ▼
Benchmark
        │
        ▼
Regression Test
        │
        ▼
Deploy new version

This is also the working method adopted by many mature Agent teams: the production environment continuously accumulates real data, constantly enriches the Benchmark, and every version upgrade undergoes a complete Regression Test to ensure the new version solves old problems without introducing new ones.

Final One-Minute Summary

If I had to summarize in one minute, I would answer like this:

A production-grade Intent Routing system is not finished once the intent is recognized, but requires establishing an engineering closed loop for continuous optimization. First, through Traces, completely record every Routing decision process to ensure problems can be quickly replayed and located; second, through Observability, continuously monitor the system's running status to detect anomalies in time; then periodically sample real data from online, combine human annotation or LLM-as-a-Judge to establish offline Evaluation, objectively assessing model capability; finally, maintain a Golden Dataset and Benchmark, and perform Regression Tests every time the Prompt or model is upgraded, ensuring the system's capability continuously improves rather than degrades. I believe a truly excellent Intent Routing is not just a single classification, but an engineering system capable of continuous evolution.

Finally, end the interview with this paragraph

If I had to summarize in one minute, I would answer like this:

The most basic solution for Agent intent recognition is to hand all Intents and their Descriptions to the LLM for classification. This method is simple to implement and suitable for Demos or scenarios with a small number of Intents.

But in a real production environment, I prefer to adopt a multi-stage Intent Routing architecture: first, quickly handle deterministic requests through rule matching; for unmatched requests, use Embedding for semantic recall, narrowing dozens or even hundreds of Intents down to a small number of candidates; finally, hand it to the LLM for precise classification, outputting Intent, Confidence, and Reason. For low-confidence scenarios, don't let the model continue guessing, but actively initiate Clarify to confirm the real intent with the user, avoiding incorrect routing.

In addition, I would establish a complete engineering closed loop. Online, record the entire routing decision process through Traces, and combine it with Observability to continuously monitor system running status and detect anomalies in time; offline, periodically build an Evaluation Dataset based on real Traces from the production environment, combined with human annotation or LLM-as-a-Judge, continuously evaluate model effectiveness, and accumulate a Golden Dataset and Benchmark. Before each upgrade of the Prompt, Embedding, or model version, perform a complete Regression Test to ensure system capability continuously improves rather than degrades.

So, I believe Agent intent recognition is not a simple Prompt Engineering problem, but an engineering system design problem integrating Routing, Observability, Evaluation, and Benchmark. A truly excellent Intent Routing is not just about letting the model recognize an intent once, but about establishing a closed-loop system capable of continuous observation, continuous evaluation, and continuous optimization.

When you answer to this point, what the interviewer hears is no longer a person who "can call a large model API." But an engineer who has truly participated in Agent system design and knows how to polish an Intent Router from a Demo into a production-grade system.

Comments

Top 1 from juejin.cn, machine-translated. The original thread is authoritative.

程序吗喽

Brilliant