跪拜 Guibai
← All articles
Frontend · Backend · Agent

Don't Blame the Prompt: Trace the First Mismatch in Your Agent's Execution

By Sailing ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Teams waste cycles bloating prompts and swapping models when the real breakage sits in tool argument generation or session context handling. Trace-based layer localization plus regression tests make Agent fixes reproducible and prevent the same class of silent parameter errors from shipping again.

Summary

The final installment of a ten-part series on production Agent engineering pushes back against the reflexive move of editing the main Prompt whenever users report bad answers. The author, who has built RAG, AI coding, workflow, and harness systems, argues that model capability only sets an upper bound; reliability depends on the full harness — sessions, tools, memory, permissions, traces, and regression tests.

Using a fictional ops assistant (OpsPilot), the article walks a typical complaint — "I said production but it queried test" — through a structured Trace. The session and intent both said `prod`, but the tool call argument somehow became `test`, and only the final answer claimed production data. That first mismatch sits at L3 (tool parameter generation), not at L5 (generation) or L4 (RAG). The fix: validate tool arguments before sending requests, merge session/intent values into parameters instead of trusting model JSON, and encode the scenario as a regression case.

Three common failure patterns get treatment: explanation questions wrongly triggering live tools, pronoun references losing their antecedent when history is truncated, and ambiguous resource names being auto-selected without user confirmation. For each, the article suggests tool-level descriptions, server-maintained conversation state, and explicit ambiguous result handling. It then covers hallucination checking — splitting answers into short claims and verifying each against tool returns — and insists every fixed incident becomes a regression case, run both automatically and manually before release. A follow-up practical demo using LangChain/LangGraph is promised.

Takeaways
— The first sign of a wrong answer often lies in tool call parameters, not in the final reply text; checking `traceId` and comparing `context`, `intent`, and `toolCalls[].args` reveals the actual failure point.
— A `status: "success"` from a tool does not mean the returned data matches the user's environment or that the answer is correct; the example shows a `test` parameter returning success while the answer claims production logs.
— Use L1–L5 (SSE/HITL, session/auth, orchestration/tools, RAG, generation/prompt) as a bug-triage vocabulary to find the first mismatched layer and avoid fixing the wrong component (e.g., skipping L4 when no retrieval occurred).
— For environment mismatches, merge the session-confirmed environment with the model's intent and validate arguments before calling the tool; reject the call if they differ and report the reason back to the user.
— When a user asks "what does this feature do?", route to knowledge-base or text answers, never to real-time tools; write this boundary into the tool description, not just the main Prompt.
— Multi-turn context loss (e.g., "its logs?" losing the service name) is fixed by sending the full relevant turns and/or maintaining a server-side `ConversationState` with confirmed service and environment; regression tests must cover complete dialogues, not just the last sentence.
— If several resources match a name, return an 'ambiguous' status and let the user choose; never auto-select the first or most similar candidate for monitoring or data queries.
— To check hallucinations, split the answer into short claims and verify each against tool returns; for example, `health: 'ok'` does not support "service recovered" because that claim needs before-and-after evidence.
— Regression cases should store the original user input, the layer where the error occurred, and the expected behavior; run them both via scripted Trace assertions and a manual spot-check before release.
— Avoid changing model, Prompt, tool, and retrieval simultaneously before a release; you won't know which change fixed the problem if several are bundled together.
Conclusions

The article's insistence on "find the first mismatch" rather than debugging from the final answer treats Agent debugging like network stack tracing: the symptom appears far downstream from the actual fault, and only a structured execution log makes the divergence visible.

The L1–L5 scheme is explicitly admitted to be team-internal notation, not an industry standard; that suggests Agent observability practices are still in a pre-standardization phase where each organization constructs its own reliable vocabulary.

The author's claim that "tool success doesn't mean business correctness" targets a common blind spot in evaluators: they check whether a function returned without error, not whether the returned data satisfies the user's stated environment or resource.

By requiring a regression case for every fixed incident and warning against mixing changes, the article frames Agent reliability as a cumulative engineering discipline rather than a one-time model upgrade — a stance that will resonate with teams that have watched prompt tweaks regress silently.

The division of intents into explanation vs. execution, and the rule to ask before auto-selecting ambiguous resources, reflects a practical stance: if the user hasn't clearly specified, the Agent should pause rather than guess, trading a little friction for far lower downstream failure rates.

Concepts & terms
L1–L5 layers
A debugging taxonomy used in this article to classify failures in an Agent system: L1 covers SSE and HITL, L2 session and permissions, L3 orchestration and tools, L4 RAG retrieval, L5 model generation and prompt design.
Trace
A structured record of a single Agent execution, including user input, session context, intent, tool call parameters, tool return summaries, and answer claims with evidence references; used to find the first divergence from expectations.
Tool Runtime
The layer that validates and executes tool calls; this article recommends adding consistency checks before sending requests so that mismatched parameters are rejected instead of silently returning successful but wrong data.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗