跪拜 Guibai
← Back to the summary

How AI Agents Keep 400 Tools From Eating the Context Window

When Tools Fill the Context Window? A Deep Dive into the Implementation of On-Demand Loading for AI Agent Tool Search

This article explains the tool search mechanism in AI Agents—how to load the tools a model needs on demand when the number of tools is so large that it fills up the context window. The content covers implementation principles, matching algorithms, caching effects, and is explained with source code from x-code-cli.

Tools and MCP Tools

LLMs Cannot Execute Operations on Their Own

Large Language Models (LLMs) are pure text generators. They can only receive text input and output text responses; they cannot actively execute external operations.

The reason an AI Agent can do these things is that the model can perform these operations through tool calls. For example, the model outputs an instruction—"I want to call the readFile tool, the parameter is package.json"—the client (here referring to the MCP client, i.e., an AI Agent application that implements the MCP protocol, such as x-code-cli, Claude Code, Cursor, etc.) code will parse this instruction, execute the actual file reading operation, and then pass the read content back to the model. After receiving the file content, the model continues reasoning and decides what to do next.

These operations that can be called by the model—reading files, executing commands, searching code, querying databases, searching the web—are collectively referred to as Tools. Each tool consists of three parts: a name (name), a natural language description (description, telling the model what this tool can do), and a JSON Schema (inputSchema, defining what parameters the tool accepts). Take x-code-cli's readFile tool as an example:

{
  "name": "readFile",
  "description": "Read the contents of a file at the specified path.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "path": {
        "type": "string",
        "description": "The absolute path of the file to read"
      },
      "offset": {
        "type": "number",
        "description": "Line number to start reading from (1-based)"
      },
      "limit": {
        "type": "number",
        "description": "Maximum number of lines to read"
      }
    },
    "required": ["path"]
  }
}

These three parts are serialized and placed into the tools array of each API request. The model determines when to use this tool based on the description, and generates valid parameters based on the inputSchema. The more complex a tool's definition (more parameters, longer description), the more tokens it consumes.

Below is the execution flow of the Agent Loop in x-code-cli:

sequenceDiagram
    autonumber
    actor U as User
    participant C as Client (x-code-cli)
    participant LLM as LLM API

    U->>C: "Help me rename the foo function in utils.ts to bar"
    C->>LLM: User message + Tool list (readFile, edit, shell, grep...)
    LLM-->>C: tool_call: readFile({path: "src/utils.ts"})
    Note over C: Client executes readFile, reads file content
    C->>LLM: tool_result: "export function foo() { ... }"
    LLM-->>C: tool_call: edit({path: "src/utils.ts", old: "foo", new: "bar"})
    Note over C: Client executes edit, replaces string
    C->>LLM: tool_result: "File modified"
    LLM-->>C: Text reply: "Renamed foo to bar"
    C->>U: Display reply

Throughout the process, the model autonomously decides what tool to call and what parameters to pass. The client is only responsible for execution and passing results. This loop may execute once or many times, mainly depending on the complexity of the task.

The Growth in the Number of Tools

Built-in Tools

When an Agent product is first launched, the number of built-in tools may not be large. However, as the product iterates, built-in tools gradually increase. For example, tools for web search, web scraping, sub-agent delegation, to-do management, background task management, etc. Mainstream AI Agent products like Claude Code, Codex, and Cursor have all gone through this growth process. It is normal for a mature AI Agent product to have dozens of built-in tools.

MCP Tools

MCP (Model Context Protocol) is an open protocol that allows Agents to access external services. For example, by connecting to a GitHub MCP server, an Agent can create issues and merge PRs; by connecting to a database MCP server, an Agent can execute SQL queries.

Each MCP Server registers all its publicly available tools with the client through the tools/list interface; this interface uses a full-return mode. After establishing a connection with a server, the client can obtain the complete list of tool definitions from that server at once.

sequenceDiagram
    participant C as MCP Client (AI Agent)
    participant S1 as MCP Server A (Exchange)
    participant S2 as MCP Server B (Database)

    C->>S1: Establish connection
    C->>S1: tools/list
    S1-->>C: Returns 400 tool definitions
    C->>S2: Establish connection
    C->>S2: tools/list
    S2-->>C: Returns 15 tool definitions
    Note over C: Client gets the complete tool list of all servers<br/>Total 415 tools, all placed into subsequent LLM requests

Structurally, MCP tools and built-in tools are no different; both are name + description + JSON Schema. The only difference is who provides them: built-in tools are hardcoded in the product code, while MCP tools are dynamically registered by external servers.

The number of tools provided by a single MCP server often far exceeds that of built-in tools. Take an MCP server for a cryptocurrency exchange as an example; it covers spot trading, contract trading, asset transfers, market queries, balance queries, and other operations, totaling 400 tools. If the user also connects some other MCP servers, the total number of tools can easily exceed 400.

Problems Caused by an Increasing Number of Tools

If the large model needs to call these tools, all tool definitions must be sent along with the request to the model. Since the large language model itself is stateless, the complete tool list must be carried anew each time an inference request is initiated.

A moderately complex tool definition is about 150–300 tokens (the name is a few tokens, the description is one or two lines, the property names and type descriptions in the JSON Schema are the bulk). 400 tools × an average of 160 tokens ≈ 64,000 tokens. Before the user even starts asking questions, the tool definitions alone occupy 64k tokens.

Let's look at the request structure when calling the LLM in x-code-cli. Below is the core code (simplified) of the runTurn function in packages/core/src/agent/loop.ts:

result = streamText({
  model,                     // LLM model instance
  system: cached.system,     // System Prompt, about 2-5k tokens
  messages: cached.messages,  // Messages (conversation history + user message)
  tools: cached.tools,       // Tools (tool definitions) ← 64k tokens when fully injected
  maxRetries: 3,
  abortSignal: options.abortSignal,
  maxOutputTokens: getMaxOutputTokens(options.modelId),
  providerOptions: mergedProviderOptions,
})

The input for a single API request consists of three fields: system, tools, and messages. The token distribution is roughly as follows:

Token Composition of a Single API Request:
┌────────────────────────────────────┐
│ system (System Prompt)             │  About 2-5k tokens
├────────────────────────────────────┤
│ tools (Tool Definitions)           │  ← When fully injected: 64k tokens (400 tools)
├────────────────────────────────────┤
│ messages (Conversation History + User Message) │  Varies with conversation length
└────────────────────────────────────┘

If the model's context window is 128k, the tool list occupies nearly half.

One point needs clarification regarding costs: mainstream LLM APIs have prefix caching. For the system + tools prefix parts that remain unchanged each round, the first request is billed at full price, and subsequent requests that hit the cache are billed at a discounted price (usually 10%–50% of the original price), so you don't pay full price every round.

But even if a 64k token tool list hits the cache, it still incurs ongoing costs: it occupies physical space in the context window, increases Time to First Token (TTFT), and squeezes the space left for conversation history and user messages. The larger the tool list, the smaller the window available for actual conversation.

This leads to the topic of this article—tool search.

Tool Search

Tool search is a built-in tool for "loading tool definitions on demand." By default, the complete definitions (name + description + JSON Schema) of all tools are placed into the tools array of each request. Tool search changes this strategy: only a list of tool names is placed in the system prompt, without the full definitions; when the model determines it needs to call a tool, it first loads the tool's complete schema via the built-in toolSearch tool, and then executes the call in the next round.

The model still knows what tools are available—the system prompt lists all tool names. But the complete definition (description + JSON Schema) of each tool is not pre-loaded into the request; instead, it is loaded only when the model actually wants to call it. This reduces the context usage from a full 64k to just 3-5k for the tool name list, plus the schemas of the few tools loaded on demand.

The Deferred Mechanism

"Deferred" literally means "delayed." In the context of tool search, a deferred tool is a tool whose loading is delayed—its complete definition (description + JSON Schema) is not sent with the request; it is only loaded when the model actively requests it via toolSearch. The counterpart is directly loaded tools, which are core tools whose complete schema is carried in every request round.

Let's look at how on-demand loading is specifically implemented, mainly involving the following steps:

Step 1: The client gets all tools from the MCP server as usual. The tools/list interface still returns the full set. When the client starts, it pulls the complete definitions of all tools from each MCP server—name, description, JSON Schema—and stores them all in local memory. This step is exactly the same as without tool search.

Step 2: Only tell the model the tool names. The complete tool schemas are not placed into the request's tools array. Tool names are grouped by server and written in the ## Deferred Tools section of the system prompt. The model can see this name list every round, knowing what tools are available, but cannot see the parameters each tool accepts.

Below is the actual ## Deferred Tools section generated by x-code-cli (excerpt):

## Deferred Tools
The tools below are available but NOT loaded — only their names are listed,
with no schema. To use one, first call `toolSearch` (keyword search, or
`select:<exact_name>` to load specific tools by name); its schema is then
added to your tool set and it becomes directly callable on your next step.
Core tools (readFile, writeFile, edit, shell, grep, glob, listDir, task)
are always loaded — never search for those.

### Built-in
- webSearch, webFetch, todoWrite, activateSkill

### Server: gate-mcp
- mcp__gate__cex_spot_get_ticker, mcp__gate__cex_spot_create_order,
  mcp__gate__cex_spot_get_balance, mcp__gate__cex_futures_get_ticker,
  ... (400 tools in total)

### Server: github
- mcp__github__create_issue, mcp__github__list_pull_requests,
  mcp__github__merge_pull_request, ...

As can be seen, this text contains two parts of information: first, usage instructions (telling the model how to load tools via toolSearch), and second, a list of tool names grouped by source. The model can see this list every round but cannot see any tool's parameter definitions.

Step 3: Give the model a built-in toolSearch tool. When the model needs to call a deferred tool, it first calls toolSearch, passing in keywords or a tool name. The client performs a match on the in-memory tool catalog and adds the matched tools to the tools array for the next request round. The model can then call it in the next round just like any normal tool.

sequenceDiagram
    participant S as MCP Server
    participant C as Client (MCP Client)
    participant LLM as LLM

    Note over S,C: At startup
    C->>S: tools/list
    S-->>C: Complete definitions of 400 tools
    Note over C: All stored in memory

    Note over C,LLM: User asks: "Help me check the latest price of BTC/USDT"
    C->>LLM: Core tool schemas + toolSearch schema + deferred name list
    Note over LLM: Sees cex_spot_get_ticker in the list,<br/>but without its schema, cannot call it directly
    LLM-->>C: toolSearch({query: "select:mcp__gate__cex_spot_get_ticker"})
    Note over C: Matches in memory catalog, adds this tool to the activated set
    C->>LLM: Core tools + toolSearch + cex_spot_get_ticker (just activated schema)
    LLM-->>C: cex_spot_get_ticker({currency_pair: "BTC_USDT"})

A key point to emphasize: deferred is something the client does. The MCP protocol's tools/list still returns the full set; the protocol itself has no capability for on-demand discovery. The deferred mechanism is entirely the client, after obtaining the full tool list, deciding on its own which ones to put into the request and which to delay loading—this is a client-side strategy, not a protocol feature.

The Model Only Sees Names, How Does It Find the Right Tool?

If only the tool name is provided without the corresponding Schema, the model cannot determine the tool's input parameter structure. To solve this problem, the system has designed multiple layers of verification mechanisms.

Tool names themselves already possess sufficient semantic description capability. Names like mcp__github__create_issue, cex_spot_get_ticker allow the model to understand the function at a glance. MCP tool names generally follow the naming convention of server__action or server__resource__verb, making the name itself a layer of semantic index. If tool names were all tool_001, tool_002, this scheme would not work.

The name list is placed where the model can see it. The ## Deferred Tools section of the system prompt lists all deferred tool names grouped by server. The model is not searching blindly—it sees the complete list, just not the parameter definitions for each tool.

The client completes the matching computation for toolSearch. toolSearch supports two calling methods: keyword search and exact match.

Method 1: Keyword Search. Used when the model is unsure of the exact tool name. The client performs keyword matching between the query parameter and the preprocessed text of each tool, sorts by score, and returns the top few.

Let's illustrate with an example. Suppose the deferred catalog has these tools:

Tool Catalog (search text preprocessed for each tool upon registration):

Tool Name                          Search Text (name tokenization + description + schema property names)
─────────────────────────────────  ────────────────────────────────────────
mcp__gate__cex_spot_get_ticker     gate cex spot get ticker Get spot market data currency_pair
mcp__gate__cex_spot_create_order   gate cex spot create order Create spot order currency_pair side amount price
mcp__gate__cex_spot_cancel_order   gate cex spot cancel order Cancel spot order order_id
mcp__github__create_issue          github create issue Create issue title body labels

After the model passes toolSearch({query: "spot ticker"}), the client scores each tool:

query = "spot ticker"

mcp__gate__cex_spot_get_ticker:
  "spot"   → name tokenization has "spot"     → +10
  "ticker" → name tokenization has "ticker"   → +10
  Total score = 20 ✓

mcp__gate__cex_spot_create_order:
  "spot"   → name tokenization has "spot"     → +10
  "ticker" → not in name tokenization, not in search text → +0
  Total score = 10

mcp__gate__cex_spot_cancel_order:
  "spot"   → name tokenization has "spot"     → +10
  "ticker" → not in name tokenization, not in search text → +0
  Total score = 10

mcp__github__create_issue:
  "spot"   → none → +0
  "ticker" → none → +0
  Total score = 0

Results (sorted by score descending):
  1. mcp__gate__cex_spot_get_ticker   (20)  ← Returned
  2. mcp__gate__cex_spot_create_order (10)  ← Returned
  3. mcp__gate__cex_spot_cancel_order (10)  ← Returned
  4. mcp__github__create_issue        (0)   ← Filtered out

Codex's search logic is similar, also pre-generating search text for each tool, but uses the BM25 algorithm for scoring (a classic ranking algorithm in the field of information retrieval, used by search engines to rank search results). The core idea is the same: match the query's keywords against the tool's search text and sort by relevance.

Neither method requires embeddings or calling a model; both are string matching—the same query, the same tool set, will yield exactly the same results.

Additionally, a keyword search may hit multiple tools simultaneously. toolSearch takes the top N by score (x-code-cli defaults to 5), and all matched tools are activated—added to the activated set, all appearing in the tools array in the next round. After the model gets the schemas of these tools, it decides on its own which one to use; the client does not restrict it.

For example, if the user says "Help me place a limit order to buy BTC," the model calls toolSearch({query: "spot order"}), which might simultaneously hit cex_spot_create_order, cex_spot_get_ticker, cex_spot_cancel_order, and other tools. The cost of activating a few more tools is a few more schemas' worth of tokens, but it's much better than fully injecting 400 tools. Moreover, activation is persistent—once activated, there's no need to search for the same tool again in subsequent rounds; it can be used directly.

Method 2: select: Exact Match. The model directly passes select:tool_name, and the client performs an exact lookup by name, no scoring needed.

How does the model know the exact tool name? Because the ## Deferred Tools section of the system prompt lists the complete names of all deferred tools. The model sees this list every round; as long as it finds the target tool name in the list, it can directly use select: to load it.

When to use which method? This logic is written in the description of the toolSearch tool itself, which the model reads before calling:

Pass `query` as either:
- keywords describing the capability you need
  (e.g. "search the web", "github create issue")
  → returns the best-matching deferred tools, or
- "select:<name>,<name>" to load specific tools by their
  EXACT name from the Deferred Tools list
  (prefer this when you already know the name).

That is, the tool description directly tells the model: If you already know the name, prefer select:; when unsure of the name, use keywords to describe the capability. In practice, the model uses select: most of the time, as it can directly see the tool names from the list. Keyword search is a fallback path for when the model is uncertain about the tool name—for example, when the tool name is long or there are multiple similar tools, the model might use keywords like "spot order" to let the client filter for it.

Comparison of the two calling methods:

Keyword Search (used when unsure of the tool name):
  toolSearch({query: "spot ticker"})
  → Client scores all tools by keyword, returns the top few by score

Exact Match (used when the tool name is known, recommended method):
  toolSearch({query: "select:mcp__gate__cex_spot_get_ticker"})
  → Client looks up directly by name, returns if found, no scoring

Keyword search is a fallback mechanism. The model can usually find the target tool name directly from the list and use select: for an exact match. Only when the model is unsure which specific tool to use does it fall back to keyword search to let the client filter for it.

Which Tools Should Be Deferred, Which Should Be Directly Loaded

Not all tools need to be deferred; some core tools are directly loaded, while tools used only occasionally are deferred.

Directly Loaded Tools (complete schema carried in every request round):

These tools are the Agent's basic operational capabilities, potentially needed for any user question:

Tool Function Why it must be directly loaded
readFile Read file Almost all coding tasks need to read files first
edit Edit file Core operation for writing code
writeFile Write file Create new files
shell Execute command Run tests, install dependencies, git operations
grep Search code Locate code positions
glob Find files Find file paths
task Sub-agent Delegate sub-tasks
toolSearch Tool search The sole entry point for loading deferred tools

Deferred (lazy-loaded) Tools:

Let's use a concrete example to illustrate the classification process. Suppose the Agent has registered the following tools:

Classification Example:

readFile        → Core built-in tool, used for every task    → Directly loaded
edit            → Core built-in tool, used for every task    → Directly loaded
shell           → Core built-in tool, used for every task    → Directly loaded
toolSearch      → Loading entry point, must be directly available       → Directly loaded
webSearch       → Non-core built-in, used occasionally         → Deferred
webFetch        → Non-core built-in, used occasionally         → Deferred
mcp__gate__cex_spot_get_ticker    → MCP tool  → Deferred
mcp__gate__cex_spot_create_order  → MCP tool  → Deferred
mcp__github__create_issue         → MCP tool  → Deferred
...（Remaining MCP tools all deferred）

Below is the flowchart of the judgment logic:

image.png

Complete Cycle: Search → Activate → Call

Let's look at a concrete example, using "check BTC price" to demonstrate the actual calling process:

sequenceDiagram
    autonumber
    actor U as User
    participant C as Client
    participant LLM as LLM

    U->>C: "Help me check the spot price of BTC"
    C->>LLM: User message + Core tools + toolSearch + deferred name list
    Note over LLM: Sees cex_spot_get_ticker in the list
    LLM-->>C: toolSearch({query: "select:mcp__gate__cex_spot_get_ticker"})
    Note over C: Exact match hit, added to activated set
    C-->>LLM: tool_result: "Loaded 1 tool(s) — now callable directly on your next step"
    Note over C: Next round's tools array appends the complete schema of cex_spot_get_ticker
    C->>LLM: Core tools + toolSearch + cex_spot_get_ticker (activated)
    LLM-->>C: cex_spot_get_ticker({currency_pair: "BTC_USDT"})
    Note over C: Execute MCP tool call, get price
    C->>LLM: Tool execution result
    LLM-->>C: "BTC spot price is 63,521.30 USDT"
    C->>U: Display reply

Note between step 3 and step 5: The model called the toolSearch tool in the first round, but cannot call cex_spot_get_ticker in the same round—because this tool does not yet exist in the first round's tools array. It must wait for the client to add its schema, and the model can only call it in the next round.

This is the cost of the deferred scheme: the first use of any deferred tool requires at least 2 rounds of API calls—first search, then call in the next round.

State Changes Throughout the Process

From startup to multi-turn conversation, the change process of various state fields:

Phase State Changes
Startup catalog = Complete definitions of 400 deferred tools obtained from all MCP servers
activated = Empty set
systemPromptCache = System prompt, containing the ## Deferred Tools name list
baseTools = [readFile, edit, shell, grep, ..., toolSearch]
Round 1 Request tools = composeTurnTools(baseTools, activated) = baseTools (activated is empty)
User: "Check BTC price" Sent to LLM: system + tools (9 core tools) + messages
Round 1 Response LLM returns: toolSearch({query: "select:mcp__gate__cex_spot_get_ticker"})
handleToolSearch() performs exact match in catalog
activated = { "mcp__gate__cex_spot_get_ticker" }
Returns tool_result to model: Loaded 1 tool(s) — now callable directly on your next step:
- mcp__gate__cex_spot_get_ticker: Get spot market data
Round 2 Request tools = [...baseTools, complete schema of cex_spot_get_ticker] (one more tool appended)
messages appended with previous round's tool_call + tool_result
Sent to LLM: system + tools (10 tools) + messages, prefix cache misses once
Round 2 Response LLM returns: cex_spot_get_ticker({currency_pair: "BTC_USDT"})
Client executes MCP tool call, gets price. activated unchanged
Round 3 tools same as Round 2 (activated unchanged, prefix stable, cache hit)
LLM returns text reply
Subsequent Rounds activated persists, no need to search for cex_spot_get_ticker again
If a new tool is activated → activated grows → tools appended → cache misses once more

A few key points:

Engineering Details

The previous section covered the core process of tool search. This section mainly discusses a few problems encountered during implementation and their solutions.

Threshold Judgment: When to enable defer. When there are few tools, full injection is simpler and doesn't require an extra round trip. x-code-cli uses DEFERRAL_THRESHOLD_PERCENT for judgment: serialize the schemas of all candidate deferred tools to estimate the token count; if it does not exceed 10% of the model's context window, skip defer and fall back to full loading.

Model Compatibility. Tool search relies on the model actively calling toolSearch. If the model doesn't support or cooperate, deferred tools will never be loaded. Different products handle this differently:

Byte Stability of the System Prompt. The prefix caching of the LLM API relies on the request prefix remaining unchanged. If the prefix composed of system + tools remains consistent in subsequent requests, it can hit the cache and be billed at a discounted price. This requires that the deferred list in the system prompt cannot be dynamically modified during runtime. x-code-cli's approach is to generate systemPromptCache at startup and freeze it, not changing it for the entire session.

This brings a side effect: after a tool is activated, the list in the system prompt still says this tool is in a deferred state, but its complete schema is already in the tools array. There is an inconsistency between the system prompt the model sees and the actual tool list: the system prompt says the tool is not loaded, but the tool is actually already in the list.

image.png

In practice, this inconsistency rarely causes problems. Strong models (Sonnet, GPT-4o level) will remember that the tool has been loaded after receiving the first toolSearch tool_result, and will call it directly in subsequent steps without repeating the search. Repeated searches only occur in extreme cases—for example, when there are very many conversation turns, early tool_results have been compressed away, and the model sees the system prompt saying "not loaded" again, it might search once more. x-code-cli adds a layer of defense: if the searched tool is already in the activated set, it returns Already loaded — call xxx directly now., preventing the model from falling into a search loop. This is defensive code, not the regular path.

Claude Code and Codex avoid this inconsistency architecturally. Claude Code relies on Anthropic API's tool_reference mechanism, where activated tools have their schemas expanded by the server side, and the client does not modify the tools array. Codex relies on the tool_search_output history item of the OpenAI Responses API, similarly handled by the server side. Neither needs to maintain a potentially stale state in the system prompt. Because x-code-cli needs to work across multiple providers, it cannot rely on the server-side capabilities of a single API and can only go through client-side re-injection—but from actual results, this inconsistency almost never triggers repeated searches under strong models.

Sub-agents do not enable defer. The tool set of a sub-agent is controlled by a whitelist; most sub-agents only have a few built-in tools. For example, the explore sub-agent only has five tools: readFile, glob, grep, listDir, shell. The plan sub-agent has even fewer, only four read-only tools. With so few tools, adding tool search is meaningless. The only exception is the general-purpose sub-agent—it inherits the MCP tools of the parent session, and the number of tools might be large, but even so, the sub-agent currently uses full injection, because the sub-agent's lifecycle is short with few turns, and the savings from defer are not significant.

Summary

A review of the complete process of tool search: At startup, tool definitions are fully obtained from MCP servers and classified into directly loaded and deferred categories based on usage frequency. For directly loaded tools, the complete schema is placed into the tools array of each request round; for deferred tools, only the names are listed in the system prompt. When the model needs a deferred tool, it calls toolSearch to load its schema, and can call it normally in the next round.

Comparison of implementations across several mainstream AI Agent CLIs:

Full Injection? Dynamic Mechanism Matching Algorithm
Claude Code No ToolSearch tool Keyword scoring + select:
Codex No tool_search tool BM25 full-text search
Gemini CLI Yes None —
x-code-cli No toolSearch tool Keyword scoring + select:

Claude Code, Codex, and x-code-cli share almost the same approach, differing in how the activated schema is made available to the model—Claude Code and Codex rely on the server-side capabilities of their respective APIs, while x-code-cli relies on client-side re-injection (directly adding the schema to the tools array). Gemini CLI currently does not have built-in tool search.

My Other AI Agent Articles

References