AI Programming · Frontend · Backend

How AI Agents Keep 400 Tools From Eating the Context Window

By 谭sir · Jul 6, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Agent tool counts are ballooning as MCP servers expose hundreds of domain-specific functions. Without on-demand loading, context windows fill with tool definitions instead of conversation, driving up latency and cost while shrinking the effective working memory available for actual problem-solving.

Summary

When an AI coding agent connects to multiple MCP servers, the tool list can easily hit 400 entries. Sending every tool's full JSON Schema with every API call burns 64,000 tokens before a single user message arrives—half a 128k context window gone. Tool search solves this by keeping only a name list in the system prompt and loading a tool's schema only when the model requests it.

The mechanism splits tools into two tiers. Core tools like readFile, edit, and shell ship with every request because nearly every task needs them. Everything else—web search, GitHub operations, hundreds of exchange-trading functions—sits in a deferred catalog. The model sees the names, calls a built-in toolSearch function with keywords or an exact select: match, and the client injects the full schema into the next round's tools array.

Claude Code, Codex, and x-code-cli all implement this pattern with minor differences in matching algorithms and how activated schemas reach the model. The trade-off is one extra API round-trip the first time a deferred tool is used, but activated tools persist for the session. A threshold check prevents the overhead from kicking in when the tool list is small enough to fit comfortably.

Takeaways

— 400 MCP tools serialized as full JSON Schema consume roughly 64,000 tokens per request, occupying half of a 128k context window before any user message.

— Tool search replaces full schema injection with a name-only list in the system prompt, cutting tool overhead to 3,000–5,000 tokens.

— The model calls a built-in toolSearch function with keywords or an exact select:<name> query; the client matches against an in-memory catalog and injects the schema in the next round.

— Keyword matching uses simple token scoring (x-code-cli) or BM25 (Codex)—no embeddings or model calls needed.

— Activated tools persist for the session, so the extra round-trip cost is paid only once per tool.

— Core tools like readFile, edit, and shell are never deferred because nearly every task requires them.

— A 10% context-window threshold check disables deferral when the tool list is small enough to fit without the overhead.

— Sub-agents skip tool search entirely since their tool sets are small and their lifecycles are short.

— Claude Code and Codex handle activated-schema injection server-side via API-specific mechanisms; x-code-cli does it client-side by appending to the tools array.

— System prompt byte stability is frozen at startup to preserve LLM prefix caching; a stale deferred list rarely confuses strong models.

Conclusions

Tool search is a client-side strategy, not an MCP protocol feature. The protocol's tools/list still returns everything; the client unilaterally decides what to defer, which means every Agent CLI must build its own implementation.

The semantic quality of tool names is load-bearing. Names like mcp__github__create_issue carry enough meaning for the model to pick the right tool from a flat list; if tools were named tool_001 through tool_400, the whole scheme collapses.

x-code-cli's model-compatibility check is a blunt string match on names like 'haiku' or 'nano', which can't distinguish Claude 3 Haiku from the far more capable Claude 3.5 Haiku. Codex's per-version boolean flag in models.json is the more precise approach.

The stale deferred list in the frozen system prompt creates a subtle state inconsistency—the prompt says a tool isn't loaded when it is—but strong models rarely trip over it because they remember the tool_result from the activation round.

Prefix caching economics shift with tool search: the cache misses only when a new tool is activated, then stabilizes. Without deferral, the cache is stable but the baseline cost is permanently higher.

Concepts & terms

Tool Search

A built-in Agent tool that loads deferred tool schemas on demand. Instead of sending every tool's full definition with each API request, only a name list appears in the system prompt. The model calls toolSearch when it needs a tool, and the client injects that tool's JSON Schema into the next round's tools array.

Deferred Tools

Tools whose complete definitions (description and JSON Schema) are not sent with every request. Only their names are listed in the system prompt. They are loaded via toolSearch when the model determines it needs them, as opposed to directly loaded core tools that ship with every request.

MCP (Model Context Protocol)

An open protocol that lets AI Agents connect to external services. An MCP server exposes tools via a tools/list endpoint; the client fetches all tool definitions at startup and can then invoke them on behalf of the model.

Prefix Caching

An LLM API optimization where the unchanging prefix of a request (system prompt plus tools) is cached after the first call. Subsequent requests with the same prefix are billed at a discount. Tool search preserves this by freezing the system prompt at startup.

BM25

A classic information-retrieval ranking algorithm used by search engines. Codex uses BM25 to score deferred tools against a keyword query, producing a relevance-ranked list without embeddings or model calls.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗