Detailed Implementation of Agent Context Management: From Window Engineering to Long-Term Memory

Core judgment: The context window is RAM, not a hard drive. A bigger window does not mean better results — research calls this context rot: the more tokens, the lower the model's recall precision on intermediate information. So the engineering goal of mainstream Agents is not "cram more in", but only feed the minimal high-signal token set at each step. This document breaks down how this is actually implemented in products like Claude Code, Cline, Aider, LangGraph, and OpenAI.

Temporarily unable to display outside Feishu document.

1. Why context management is the first problem in Agent engineering

Ordinary chat is one question, one answer; context clears after a few turns. An Agent is a loop — the model calls tools itself, looks at results, calls the next tool, and runs dozens or hundreds of turns per task. Each turn produces new tokens: user messages, model reasoning, tool call JSON, tool return results. The window quickly fills up.

Filling up isn't simply "an error" but four kinds of implicit failures, each of which quietly makes the Agent worse:

Failure Mode	Meaning	Typical Symptom
Context Poisoning	An early hallucination gets stuffed into history, and all later turns reason off that erroneous premise.	The Agent drifts further off track; facts it states to the user don't match.
Context Distraction	Too much irrelevant information fills the window, diluting the model's attention and losing focus.	Even when the right clue is given, the model ignores it.
Context Confusion	History contains multiple similar but different pieces of information; the model confuses which one the current task should use.	Uses the previous task's file path / parameters.
Context Clash	Two pieces of history contradict each other (user says A then says B); the model doesn't know which to follow.	Output is self-contradictory, or inexplicably picks the old value.

Combined with context rot (long-window recall precision naturally declines) and cost/latency linearly rising with tokens, the conclusion is direct: no matter how big the window (128k, 200k, 1M), active context management is mandatory. Anthropic's own phrasing — "treat context as a finite resource, not a bucket you can fill infinitely."

2. What a typical Agent's context actually consists of

Breaking down one LLM call's input, in order, it's usually this:

Component	Content	Changes across turns	Typical share
System Prompt	Role, behavior rules, output format, refusal policy, tool usage examples.	Basically unchanged	5%–10%
Tool Definitions	All available tools' name + JSON Schema + description.	Basically unchanged (unless tools are dynamically added/removed)	5%–15%
Long-term Memory / Project Files	CLAUDE.md, rules files, user profile, key fact notes.	Occasionally updated	3%–8%
Message History	Conversation history: message lists of user/assistant/tool roles.	Appended each turn	20%–40%
Tool Results	Return results of recent tool calls (file contents, search results, SQL output).	Appended each turn	15%–30%
Current User Message	This turn's user input.	Only this turn	1%–5%

Key observation: only the last 50%–70% truly changes each turn. The first 30%–40% (system + tools + memory file) is fixed across turns — this is the physical basis for Prefix Cache, covered later.

The four things about context management — Write / Select / Compress / Isolate — essentially manage each row of this table: which rows to preload, which to fetch on demand, which to compress, which to split out.

3. Strategy One: Write — put information outside the window

Core idea: only put "what this step must use" in the window; write the rest to external storage (files, databases, state objects), and read it back with tools when needed.

3.1 Scratchpad

Let the Agent maintain its own working notes, persisting across turns. Common implementations:

• File as note: Claude Code's TODO list, NOTES.md / progress.md at project root. The Agent updates with Edit/Write tools; reads back with Read next turn.
• State object field: In LangGraph, add a notes field to the state schema; the model writes it each turn and the system persists it automatically, no tool call needed.
• Dedicated Memory tool: Anthropic shipped a memory tool (beta) with Sonnet 4.5 — extracts "save note / read note" into two function calls; the model decides when to write and read.

Anthropic ran experiments with Claude playing Pokémon: without memory, it forgot what it was doing after a few hundred steps; with structured notes, the Agent itself maintained "explored-area map / unlocked items / battle strategy" in notes, and remained coherently advancing across thousands of steps.

3.2 Procedural Memory: the CLAUDE.md kind

Not facts, but "rules for doing things". Typical forms:

• Claude Code's CLAUDE.md (project root) and user-level ~/.claude/CLAUDE.md.
• Cursor's .cursorrules, Windsurf's .windsurfrules.
• Cline's Memory Bank: a set of markdown files in the project (projectBrief.md, techContext.md, productContext.md, etc.), auto-loaded at each startup.

These files are stuffed into context every turn (they belong to the "long-term memory" segment), so be restrained — they are extensions of the system prompt, not logs. Anthropic's advice is "minimal but complete": enough to guide behavior without redundancy.

3.3 Long-term Memory: cross-session user profiles

Cross-session memory of "who the user is, what they prefer, what they did before". Products that specialize in this layer:

Product	Core mechanism	Write/recall method
Mem0	Extract facts from conversation; auto-deduplicate, score, and store in a vector database.	Extract memory asynchronously after each user/assistant turn; recall via embedding + time decay.
Letta (MemGPT)	Treat the LLM as CPU, explicitly split main context (RAM) and external context (disk); model decides page in/page out itself.	Manual management via core_memory (in window) and archival_memory (external) tools.
Zep	Temporal knowledge graph: build memory as a graph (entities + relations + time).	Not simple vector retrieval but graph queries — can answer "what tech stack did the user's last-mentioned project use."
ChatGPT Auto Memory	Automatically identify facts worth remembering in conversation; store in user-level memory.	Recall via embedding; problem is false recall (Simon Willison cited an example: he asked for image generation, and the model also pulled "which city he lives in" from memory).

Engineering experience: writes should be asynchronous (don't make the user wait for memory extraction); recall should be isolated by user (A's preferences can't be used for B); outdated facts should have a TTL or be overwritten by newer facts.

4. Strategy Two: Select — only bring in what's relevant each turn

Write solves "where to store"; Select solves "what to take each turn". Not all information should enter the window.

4.1 Preloading vs on-demand loading

Strategy	Approach	When suitable
Preload (eager)	At startup, stuff system prompt, tool definitions, CLAUDE.md, and the most recent task state into the window.	Small projects, stable rules, first-turn latency insensitive.
On-demand (lazy / just-in-time)	Put only lightweight identifiers in the window (file paths, URLs, queries); model uses Read/grep/HTTP tools to fetch as needed.	Large codebases, long documents, dynamic data. Claude Code does this: CLAUDE.md is preloaded, other files are fetched on demand via glob + grep + Read.

Anthropic calls the latter "just-in-time context": don't precompute and stuff things in; give the model tools to navigate itself. Benefit: no complex index to maintain. Cost: an extra tool round-trip. Production usually mixes: stable rules preloaded, dynamic content fetched on demand.

4.2 RAG: retrieval by relevance

When a knowledge base is too large to fit in the window, do retrieval. Engineering details:

• Chunking: split by semantic boundaries (code uses AST to split functions/classes; docs by paragraphs + overlap), don't hard-split by fixed character count.
• Hybrid retrieval: two-path recall — embedding (semantic) + BM25 (keyword) — then re-rank with a cross-encoder for Top-K. Pure embedding is unreliable on codebases (Windsurf engineering blog's own words).
• Context concatenation template: retrieved results aren't directly concatenated into history; wrap them in a <context>... block placed before the user message, with source annotated, so the model distinguishes "reference retrieved" from "user's original words."

4.3 Tool RAG: when there are too many tools, select among them too

When tool count exceeds ~20, tool definitions alone can eat 10k+ tokens, and the model may select the wrong tool because descriptions are similar. Approach: embed all tool descriptions, retrieve Top-K relevant tools per turn based on user message semantics, and put only those into system. LangGraph's BigTool library does exactly this; the paper says tool selection accuracy improves ~3x.

5. Strategy Three: Compress — squeeze when the window is about to blow

The previous ones are prevention; Compress is the safety net. Trigger is usually "window used 80%–95%."

5.1 Tool Result Clearing (lightest)

The safest compression: don't change the conversation; only replace old tool results with placeholders. For example, the grep result from turn 5 is useless by turn 30; replace it with:

[Tool result cleared: grep returned 12 matches in src/; see original output at turn 5]

Anthropic's "tool result clearing" on Claude Developer Platform does this — doesn't alter model behavior, just saves tokens. This is the first tier of things to do.

5.2 Auto-Compact (full-trajectory summarization)

When the window approaches full (Claude Code hits 95%), feed the entire conversation history to the model, ask it to output a summary, then restart a new window with "summary + last N turns of raw messages."

The key is that the summary prompt must explicitly specify what to keep and what to drop. Claude Code's approach:

• Keep: architectural decisions, unresolved bugs, implementation details, user preferences, current task goals.
• Drop: redundant tool outputs, repeated reasoning processes, details of already-completed subtasks.
• Append: after compaction, append the 5 most recently accessed files so the model has immediate context.

Engineering experience: tune recall first (ensure the summary doesn't drop key facts), then tune precision (remove redundancy). Compressing too hard loses details that "look useless now but turn out critical later."

5.3 Recursive / hierarchical summarization

Very long tasks (multi-hour migrations, dozens of turns of research) won't be compressed only once. Approach:

• Every N turns, make a local summary (that segment's tool calls and results).
• When the window fills again, compress multiple local summaries into one global summary.
• Forms a tree of "summaries of summaries," similar to human meeting-note layering.

5.4 Trimming (hard cut)

The most brutal way: directly drop the oldest K messages. Advantage: zero cost, deterministic. Disadvantage: may drop early important constraints (user's first-turn "don't use X library"). Engineering generally pairs it with "system-level message protection": system prompt and the first few key user messages are never trimmed.

6. Strategy Four: Isolate — split windows, don't have one Agent carry everything

Some tasks inherently need exploration ("research A, B, C three options"), and having one Agent search all three in one window results in mutual interference. Splitting into sub-Agents is the cleanest solution.

6.1 Sub-Agent

The main Agent only plans and synthesizes; each sub-Agent takes an independent window to dive deep:

• Each sub-Agent has its own system prompt, toolset, and message history.
• It can spend tens of thousands of tokens searching, reading files, running commands.
• On completion, it returns only a 1k–2k token summary to the main Agent.

In Anthropic's multi-agent research system paper, this architecture clearly outperformed a single Agent on complex research tasks. The cost: token consumption can be 15x a single Agent — so only use it when parallel exploration is valuable.

6.2 Sandbox (isolating large objects)

CodeAgent (the kind used by HuggingFace deep researcher): model outputs a piece of Python code, executed in a sandbox. Large objects (images, audio, CSV, intermediate results) stay directly as Python variables in sandbox memory, never entering the LLM context. What's returned to the model is only a few printed lines or variable references. This is the standard way to isolate "heavy data."

6.3 State Schema isolation

LangGraph-style state objects: schema has many fields, but only messages is exposed to the LLM. Other fields (tool result cache, intermediate computation, user profile) stay in state; code decides when to inject them. So the LLM always sees a clean message stream, never polluted by intermediate state.

Temporarily unable to display outside Feishu document.

7. Implementation breakdown of mainstream products

7.1 Claude Code (Anthropic official CLI)

• Preload: project root CLAUDE.md + user-level ~/.claude/CLAUDE.md + system prompt.
• On-demand: built-in tools Read / Glob / Grep / Bash; model navigates file system itself. No codebase index.
• Compression: auto-compact triggers at 95% window; model outputs summary + last 5 files to restart.
• Memory tool: since Sonnet 4.5, a dedicated memory tool (writes files to memory directory).
• Sub-agents: can spawn sub-Agents; main Agent only takes the summary.

7.2 Cline (VS Code coding Agent)

• No codebase index: official blog explicitly explains — RAG indexing is unreliable on large codebases and goes stale. Letting the model explore on demand via Read/Grep is more accurate.
• Memory Bank: prebuilt set of markdown files in the project (projectBrief / techContext / productContext / activeContext), loaded at each session start, serving as procedural memory.
• Progressive disclosure: when tools return large files, don't stuff everything at once; paginate, truncate, show directory tree, let the model decide whether to keep reading.

7.3 Aider (terminal pair-programming)

• Repo Map: use tree-sitter to parse the entire codebase, extract each file's classes/functions/methods, generate a symbol map.
• Importance scoring: score by import relationships, reference count, distance to the currently edited file; only put top-N symbols into context (typically a few hundred lines).
• Benefit: model knows "what functions exist in the project and where they are" without reading whole files, so it doesn't hallucinate APIs when editing code.

7.4 LangGraph (orchestration framework)

• Short-term memory: checkpointer automatically persists each step's state (SQLite/Postgres), effectively an automatic scratchpad.
• Compression node: add a summarization node in the graph; a conditional edge decides "if message history exceeds N tokens, go down the compression branch."
• Long-term memory: LangMem / Store API, support embedding retrieval and per-user partitioning.
• Isolation: state schema design + subgraphs naturally isolate different phases' context.

7.5 OpenAI Responses API / Assistants

• Thread: server maintains conversation history; client only sends new messages each time, no need to manage the messages array itself.
• Auto-truncation: when exceeding model context window, server automatically drops old messages (specific strategy opaque).
• Built-in tools: web_search / file_search / code_interpreter all run server-side; large results don't come back into context.

7.6 Long-term memory specialized layers: Mem0 / Letta / Zep

When not building your own wheel, use directly: Mem0 fits "user profile + preferences," Letta fits "Agent autonomously manages memory," Zep fits scenarios needing entity-relation reasoning. They are all standalone memory services; Agent reads/writes via API, not occupying LLM context.

8. Prefix Cache engineering: the underestimated cost lever

As said earlier, system prompt + tools + memory file is fixed across turns. Mainstream inference services (OpenAI, Anthropic, Volcano Ark) all support Prefix Caching: this fixed prefix is only computed once; later turns hit cache and reuse it directly; input token price drops to 10–20% of original; first-token latency drops dramatically.

Hard engineering rules:

• Put fixed prefix first: system prompt → tools → memory file → recent messages. Order must not change.
• Don't change system prompt / tool list / thinking config within the same session — changing one invalidates the whole segment's cache. Debugging can tolerate it; production must freeze it.
• User messages and history go after the prefix, so their changes don't affect prefix cache hits.
• The cost bulk of multi-turn Agents after cache hits is actually input tokens, not output. A long session without prefix caching can differ on the bill by 5–10x.

9. How to handle tool results: the easiest part to overlook

Tool results are the largest source of context bloat — a search returns 10 results, a Read opens a 500-line file, a SQL query returns 1000 rows. Stuffing them all in the window fills it in 3 turns.

Tool type	Handling strategy
File reads	Over ~200 lines, return only directory / first N lines + file size hint; let the model decide whether to continue with offset.
Search results	Return only title + URL + snippet, not full text; re-rank then take Top-5.
SQL / API	Limit returned rows (LIMIT 20), return schema + first few sample rows, no full dump.
Command output	Keep stderr fully; truncate stdout to ~2000 tokens, attach "output truncated; use head/tail tool for more."

Core principle: what tools return to the LLM is "index and samples," not "data itself." Data stays on the tool side; if the model wants more, it calls the tool again.

10. Pre-launch context management checklist

Question	Qualification standard
What is the fixed-prefix share?	system + tools + memory file total ≤ 30% of window, and unchanged across turns (prefix-cache friendly).
How are tool results truncated?	Large results have explicit truncation strategy and a "keep reading" tool, not full dump.
What happens when window hits 80%?	Have automatic compression (tool result clearing → summarization), not hard cut or error.
What to keep in compression?	Have explicit summary prompt: keep task goals, unresolved problems, user preferences; drop redundant tool outputs.
Where does cross-session memory live?	Standalone memory service / file, isolated by user, with expiration policy.
More than 20 tools?	Do semantic retrieval on tool descriptions; only stuff relevant Top-K each turn.
How to split long tasks?	Sub-Agent with independent window, only returns summary; large objects go to sandbox, not context.
How to verify context management works?	Have offline case library; after changing compression strategy, run regression, compare task success rate doesn't decline.

Sources

• Anthropic: Effective context engineering for AI agents (2025-09): official definitions of context rot, compaction, structured note-taking, sub-agent three long-task techniques.
• LangChain: Context Engineering (2025-07): the Write / Select / Compress / Isolate four-strategy framework, plus engineering practices from Cognition, Windsurf, HuggingFace.
• Claude Platform: Context Editing (Beta): server-side API for automatically managing conversation context.
• Cline: How to Think About Context Engineering (2025-08): context trade-offs for coding Agents.
• Cline: Why We Don't Index Your Codebase (2025-05): why coding Agents don't do RAG indexing.
• Aider: Repository Understanding and Context: Repo Map's tree-sitter implementation.
• LangChain Deep Agents: Context Engineering: Deep Agents' input context / runtime context / offloading / isolation practices.
• Mem0: Long-Term Memory for Agents: production-grade long-term memory write and recall.

This article is an engineering survey; specific thresholds (like 95% auto-compact, Top-K values) come from each product's public docs and engineering blogs; real projects must adjust based on their own tasks and model windows.