跪拜 Guibai
← All articles
Artificial Intelligence · Frontend · Backend

Hand-Rolled Agents, Part 4: When Context Overflows, Trim, Summarize, or Offload

By 十有八七 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Long-running agent loops in production will hit context limits and silently degrade answer quality before crashing. The three management strategies plus a file-based memory layer give developers a pragmatic, code-level playbook for keeping an agent coherent over long sessions and across restarts, while also exposing the serious caveat that model-written memory is unreliable.

Summary

Context windows have hard token limits, but models start missing details long before the limit is hit. This phenomenon is known as context rot: the fuller the window, the more likely the model overlooks a specific instruction or fact, especially ones buried in the middle. So a hand-rolled agent needs proactive management, not just error handling.

Three strategies exist for keeping the trajectory under control. Trimming cuts what goes in at the source: tools return only fragments, old results get replaced with a pointer. Compression runs a separate model call to summarize early rounds, and swaps the summary in. Offloading writes content to external files and leaves only a one-line reference in context. Each has trade-offs: trimming risks discarding something needed later, compression costs an extra call and invalidates the prompt cache, offloading requires an extra read when the details are needed.

The post then extends the agent's memory across sessions with a single markdown file. A `remember` tool lets the model append long-term facts, and at startup the file content is appended to the system prompt. A concrete run shows the agent recalling a note-title format after a restart, but also reveals a failure: the model inferred a false rule about leading zeros and followed it. The conclusion is that memory written by the model can be wrong, so it must stay human-readable and manually editable.

Takeaways
— Context windows are finite, and models begin missing details before the hard limit is reached; this is called context rot.
— Set a token or character budget well below the model's context limit, and trigger management when the trajectory exceeds that budget.
— Trimming means making tools return less data and marking omitted content so it can be re-fetched when needed.
— Compression replaces an earlier segment of trajectory with a summary: it requires an extra model call, loses details, and invalidates the prompt cache from the summary point onward.
— Because compression invalidates cache, do it in large batches saved up over time, not a little each loop.
— Offloading writes results or progress to external files, leaving only a short pointer line in the trajectory; files have nearly unlimited capacity.
— Cross-session memory is a file plus two operations: a `remember` tool appends facts, and startup reads the file into the system prompt.
— The `compact` function splits the trajectory at the last user message, summarizing only the older part and keeping the latest round intact.
— The model wrote a memory entry that contradicted the user's actual statement, proving that model-generated memory can be wrong and must be human-readable and editable.
— Memory should only store facts useful a day later or across tasks: long-term preferences, stable facts, and recurring corrections. Exclude one-off requests, temporary state, intermediate results, and sensitive information.
Conclusions

Treating model memory as a feature is a conceptual dead end; the entire architecture of long-horizon agents hinges on what you choose to put into the context window and how you refresh it.

Compression is clever but costly in subtle ways: the cache invalidation alone forces a batched, coarse-grained approach that directly conflicts with smooth incremental updates.

The memory file example demonstrates a practical truth: when you let the model write its own memory, you inherit its hallucination tendencies permanently, so a human-in-the-loop review process is not optional.

The three escape routes form a continuum of locality: trim at the tool boundary, compress within the window, and offload beyond the process. Choosing among them is a cost trade-off, not a one-size-fits-all solution.

Concepts & terms
Context window
The maximum number of tokens a language model can process in a single request. Exceeding it causes an error.
Context rot
The degradation of model performance as the context window fills up, where details are increasingly overlooked, especially those buried in the middle.
Trajectory
The accumulated conversation history, tool calls, and tool results that an agent passes to the model on each step.
Compression
Replacing an earlier segment of the trajectory with a model-generated summary. Saves space but costs an extra model call, loses detail, and invalidates the prompt cache from the summary point onward.
Offloading
Moving content out of the context window into an external file, leaving only a short reference in context. Requires an extra read when the detail is needed.
From the discussion

The discussion centers on two concrete flaws in the hand-rolled agent's memory and trim logic. The memory file's append-only design stacks contradictory user preferences over time, forcing the model to weigh stale constraints against current ones with no way to tell recency. Trimming by entry count can exit early when the last entries alone exceed the budget, deferring the failure to the next request. A separate remark reinforces the need for human-readable memory and manual review, since model-generated rules drift over time.

— An append-only memory file accumulates outdated preferences, and since the model can't distinguish which instruction came later, conflicting rules (e.g., brief vs. detailed replies) cause it to waver between them.
— The memory file needs both deletion of individual entries and overwrite semantics per topic, so newer information replaces older instead of stacking.
— Count-based trimming can terminate the loop while the remaining entries still exceed the context budget, pushing the overflow error to a later request instead of handling it locally.
— Human-readable memory is preferable to machine-derived rules because the latter tend to drift; periodic manual inspection is necessary to prevent corrupted memories from compounding.
Featured comments
booboosui 1 likes

Section 2.3 pushes external storage one step further, and you'll hit the issue that remember only appends: memory.md only grows, never shrinks. A user first says replies should be short, then half a month later says they should be detailed. Both entries are still there, and on next startup both get stuffed into the prefix, so the model oscillates between the two preferences because it can't tell which one came later. This file at least needs two things: individual entries deletable, and new entries on the same topic overwrite old ones. Otherwise, the older entries increasingly behave like hard constraints. One more thing: if trimming stops by count (e.g., stop when fewer than four entries remain), but the last two entries themselves already exceed the budget, the loop exits directly and returns something still overlong. Locally it won't surface; the error only shows up on the next request.

组合搭配函数 1 likes

The point about storing memory as human-readable text is spot on — rules the model derives on its own really do go off the rails. Periodic manual checks are something you can't skip, otherwise wrong memories get more skewed the longer they're used.

See top comments, translated →
Source: juejin.cn ↗ Google Translate ↗ Backup ↗