跪拜 Guibai
← Back to the summary

Hand-Rolled Agents, Part 4: When Context Overflows, Trim, Summarize, or Offload

This is Part 4 of the "Hand-Rolled Agent" series, 10 parts total. Series guide and index | Previous: Opening the Eyes: Context Is the Whole World It Sees

Figure 4-1 Series map

Figure 4-1 Series map. This part is still in the blue box: eyes.

Preface

The previous part left two questions: the trajectory only grows, eventually it won't fit; and when the program is closed, it remembers nothing.

These two problems are actually the same thing: memory. I originally thought "memory" was a feature of the model, that flipping a switch would make it remember. Only after studying this did I understand the model has no such feature. As covered in Part 1, its parameters are fixed after training; every request is brand new to it. So-called memory is entirely about working with context.

This part builds both kinds of memory. Xiaoji upgrades to v0.4.

Key points of this part:

1. The Window Is Limited, and Problems Appear Before It Fills

A model has a maximum number of tokens it can look at per request, called the context window. Different models range from tens of thousands to millions. Exceeding it causes the request to error directly.

Even staying under the limit is not safe. The more content in the context, the more likely the model misses a particular piece, especially ones in the middle. This is similar to humans: reading a hundred pages of material in one sitting then answering a question often performs worse than reading just the relevant three pages. This phenomenon has a name: context rot.

So you cannot wait for an error to handle it. My approach is to set a budget noticeably lower than the limit, and act once the trajectory exceeds the budget.

Figure 4-2 Context usage

Figure 4-2 The gray line ignores it, growing until it won't fit. The blue line compresses each time the budget is hit, staying in a safe range.

Combined with the cost math from the previous part: the longer the context, the more expensive each loop. Controlling context length serves three purposes: avoiding errors, improving answer accuracy, and saving money.

2. Three Escape Routes

When the trajectory gets too long, there are three things you can do.

Figure 4-3 Three escape routes

Figure 4-3 Trimming means putting in less at the start; compression means swapping in shorter versions; offloading means moving elsewhere.

2.1 Trimming: Put Less In from the Start

From the weight check in the previous part, the largest chunk of context is tool results. The most cost-effective approach is to have tools return less from the beginning.

The cost is possibly removing something needed later. So trimming should leave a fallback path so it knows how to get the content back.

2.2 Compression: Replace Old with Summaries

Take an earlier chunk of trajectory, ask the model to write a summary, and replace the original text with it.

Figure 4-4 Before and after compression

Figure 4-4 After compression the trajectory is much shorter, but whatever the summary omitted is truly gone.

At minimum the summary must keep four things: requirements I've stated, what has been done, conclusions reached, and unfinished work. Missing any of these could throw it off track later.

Compression has three costs. First, it incurs an extra model call. Second, details are definitely lost. Third, it conflicts with the caching discussed in the previous part: since compression modifies earlier content, the cache becomes invalid from the summary point onward.

Because of the third point, compression cannot be done a little each loop, as each loop would then invalidate the cache. Instead, save up and compress a large chunk at once, which is why the blue line in Figure 4-2 is sawtooth-shaped.

2.3 Offloading: Move to Files

Move content outside the context, leaving only a pointer phrase in the context.

For example, after Xiaoji reads ten notes and produces a summary, that summary doesn't have to stay in the trajectory. Write it into summary.md, and keep only a phrase in the trajectory like "Summary written to summary.md". When detailed content is needed later, read that file.

Files have almost no capacity limit, which context can't match. In long tasks, having the Agent write progress and conclusions into files as it goes follows the same logic: even if the trajectory gets compressed later, the file content survives.

The three routes can be used together. Side-by-side comparison:

Approach Cost
Trimming Tools return less, old results omitted May remove something needed later
Compression Replace old trajectory with summary Extra call, loses detail, invalidates cache
Offloading Write content to file, keep only pointer May need an extra read when used

3. Hands On: Compress When Over Budget

Here is the compression code. Focus on how the compact function splits the trajectory into "earlier" and "recent" segments.

import json, os, pathlib
from openai import OpenAI

client = OpenAI(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"])
MODEL = os.environ["LLM_MODEL"]
BUDGET = 600   # Character budget for the trajectory. Set small for demonstration; in practice, set based on the model's window size

def as_dict(m):
    return m if isinstance(m, dict) else m.model_dump(exclude_none=True)

def compact(messages):
    """When trajectory exceeds budget, keep the latest round of exchange and replace earlier parts with a summary."""
    history = [as_dict(m) for m in messages[1:]]
    last_user = max((i for i, m in enumerate(history) if m["role"] == "user"), default=0)
    old, recent = history[:last_user], history[last_user:]
    if len(json.dumps(history, ensure_ascii=False)) <= BUDGET or not old:
        return messages
    ask = ("Below is an earlier conversation record. Please write a short summary that keeps: requirements I've stated, what has been done, conclusions reached, and unfinished work.\n\n" + json.dumps(old, ensure_ascii=False))
    summary = client.chat.completions.create(
        model=MODEL, messages=[{"role": "user", "content": ask}]).choices[0].message.content
    print(f"  [Compress] Replaced earlier {len(old)} messages with a summary")
    return [messages[0], {"role": "user", "content": "【Summary of previous conversation】\n" + summary}] + recent

Key points:

4. Cross-Session Memory: A Memory File

4.1 Store Outside, Put Back Next Time

Compression solves the problem of a single run not fitting. But the trajectory only lives within that run; once the program closes, it's gone. To still remember tomorrow, the only way is to store outside the program. The simplest is a file.

Figure 4-5 The two kinds of memory

Figure 4-5 The memory file bridges two runs.

The whole thing consists of two actions.

4.2 Memory Tool and Startup Reading

The code below does these two things. Focus on how remember is described, and where the memory is placed in context.

MEMORY = pathlib.Path("memory.md")   # Cross-session memory lives in this file

def remember(fact):
    """Append a fact worth remembering long-term to memory.md."""
    with MEMORY.open("a", encoding="utf-8") as f:
        f.write(f"- {fact}\n")
    return "Remembered."

TOOLS = [{"type": "function", "function": {
    "name": "remember",
    "description": "Record a fact that will be useful in future conversations, such as my long-term preferences or fixed habits. Do not record things useful only in this conversation.",
    "parameters": {"type": "object", "required": ["fact"],
                   "properties": {"fact": {"type": "string", "description": "One sentence stating clearly what the fact is"}}},
}}]

# At each startup, append the memory file content after the system prompt
memory_text = MEMORY.read_text(encoding="utf-8") if MEMORY.exists() else "(none yet)"
SYSTEM = "You are Xiaoji, an assistant that helps organize study notes. Keep answers short.\n\n【Long-term memory about me】\n" + memory_text
messages = [{"role": "system", "content": SYSTEM}]

4.3 Connect the Loop and Run Twice

Finally, the loop. Almost identical to the one in Part 2, except for the first line: before each new question comes in, check whether compression is needed.

def run(question, max_steps=10):
    messages[:] = compact(messages)                     # Before a new question, check if compression is needed
    messages.append({"role": "user", "content": question})
    for step in range(1, max_steps + 1):
        resp = client.chat.completions.create(model=MODEL, messages=messages, tools=TOOLS)
        msg = resp.choices[0].message
        messages.append(msg)
        if not msg.tool_calls:
            return msg.content
        for call in msg.tool_calls:
            result = remember(**json.loads(call.function.arguments))
            print(f"  [Step {step}] {call.function.name} {call.function.arguments}")
            messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
    return "Steps exhausted without finishing."

while True:
    q = input("Me: ").strip()
    if q in ("exit", "quit", "q"):
        break
    print("Xiaoji: " + run(q))

Compression is done when a new question arrives, because at that point the previous round of Q&A is complete, avoiding splitting a tool call from its result.

Put the three code blocks in order into a single file, and you get Xiaoji v0.4. I ran it twice. First, I told it a preference, then chatted a few more rounds:

Me: From now on, my note titles should use the format "date-topic", e.g. 1118-Context.
  [Step 1] remember {"fact": "User requires note titles in 'date-topic' format, e.g., 1118-Context..."}
Xiaoji: Okay, noted: title format is "date-topic", like 1118-Context. I'll use that for notes going forward.
... (chatted three more rounds)
  [Compress] Replaced earlier 4 messages with a summary

Exit, restart, then ask:

Me: Help me title today's note about memory.
Xiaoji: What's today's date? Give me month-day (e.g., May 6 → 56), and the title will be "56-Memory".

After restart it still remembers the title format. You can also see it still doesn't know today's date; the limitation from Part 1 remains.

But when I opened memory.md, I noticed it had recorded more than what I said: "Date uses four digits month-day, no leading zeros." I never said that. It inferred it from my example, and got it wrong. The "56" in the second run came from that wrong memory.

Memory is written by the model. It can misremember and add things I never said, and wrong memory keeps affecting every later conversation. So I store memory as plain text humans can read, check it occasionally, and manually fix errors.

5. What to Remember, What Not to Remember

To remember Not to remember
Long-term preferences: note titles use "date-topic" Requests that only apply this once: keep this answer short
Stable facts: I'm learning Agent, already done tool calling Temporary state: I'm currently reading paragraph 3
Recurring corrections: don't use English abbreviations Task intermediate results; that's for external files
Sensitive info like passwords, ID numbers

The rule of thumb is to ask: will this still be useful a day later or for a different task? If yes, record it.

Three problems that memory accumulates over time. Xiaoji v0.4 doesn't handle any of them; just pointing them out:

Memory management is hard not because of storage, but because of maintenance.

Common Misconceptions

I originally thought Actually
The model has memory Models don't remember by themselves. Memory is something stored externally and put back into context when needed
As long as context doesn't exceed the limit, it's fine The fuller it is, the more likely it overlooks things; set a lower budget
Compression has no loss Summaries inevitably lose detail, cost an extra call, and invalidate cache
The more memory, the better Too much memory takes up context, goes stale, and contradicts itself
Memory it writes itself is reliable It can misremember and add things I never said; it must be viewable and editable

Summary

This version of Xiaoji:

Xiaoji v0.4
What's new vs previous Auto-compress when trajectory exceeds budget; a memory file and remember tool
Capabilities Long conversations won't blow the context; remembers preferences after restart
Remaining issues Memory and notes are both read in whole; will fail when content grows

Next Part

Right now Xiaoji has only one way to find things: read everything in. The memory file is read fully at startup, and notes are read fully via read_note.

Once notes reach two hundred, this approach won't work. It can't read them all; it has to guess which ones to read based on filenames. But if I ask "why does the model forget what was said earlier" and the note is titled "context window overflow", the filename and question have zero words in common. How would it find it?

How to pull only the relevant passages from a large pile of material into context? The next part teaches it to flip through books.

Next (Part 5): Learning to Flip Through Books: RAG, Look Up Before Answering

Comments

Top 2 from juejin.cn, machine-translated. The original thread is authoritative.

booboosui 1 likes

Section 2.3 pushes external storage one step further, and you'll hit the issue that remember only appends: memory.md only grows, never shrinks. A user first says replies should be short, then half a month later says they should be detailed. Both entries are still there, and on next startup both get stuffed into the prefix, so the model oscillates between the two preferences because it can't tell which one came later. This file at least needs two things: individual entries deletable, and new entries on the same topic overwrite old ones. Otherwise, the older entries increasingly behave like hard constraints. One more thing: if trimming stops by count (e.g., stop when fewer than four entries remain), but the last two entries themselves already exceed the budget, the loop exits directly and returns something still overlong. Locally it won't surface; the error only shows up on the next request.

组合搭配函数 1 likes

The point about storing memory as human-readable text is spot on — rules the model derives on its own really do go off the rails. Periodic manual checks are something you can't skip, otherwise wrong memories get more skewed the longer they're used.