Hand-Rolled Agents, Part 4: When Context Overflows, Trim, Summarize, or Offload
Long-running agent loops in production will hit context limits and silently degrade answer quality before crashing. The three management strategies plus a file-based memory layer give developers a pragmatic, code-level playbook for keeping an agent coherent over long sessions and across restarts, while also exposing the serious caveat that model-written memory is unreliable.
Context windows have hard token limits, but models start missing details long before the limit is hit. This phenomenon is known as context rot: the fuller the window, the more likely the model overlooks a specific instruction or fact, especially ones buried in the middle. So a hand-rolled agent needs proactive management, not just error handling.
Three strategies exist for keeping the trajectory under control. Trimming cuts what goes in at the source: tools return only fragments, old results get replaced with a pointer. Compression runs a separate model call to summarize early rounds, and swaps the summary in. Offloading writes content to external files and leaves only a one-line reference in context. Each has trade-offs: trimming risks discarding something needed later, compression costs an extra call and invalidates the prompt cache, offloading requires an extra read when the details are needed.
The post then extends the agent's memory across sessions with a single markdown file. A `remember` tool lets the model append long-term facts, and at startup the file content is appended to the system prompt. A concrete run shows the agent recalling a note-title format after a restart, but also reveals a failure: the model inferred a false rule about leading zeros and followed it. The conclusion is that memory written by the model can be wrong, so it must stay human-readable and manually editable.
Treating model memory as a feature is a conceptual dead end; the entire architecture of long-horizon agents hinges on what you choose to put into the context window and how you refresh it.
Compression is clever but costly in subtle ways: the cache invalidation alone forces a batched, coarse-grained approach that directly conflicts with smooth incremental updates.
The memory file example demonstrates a practical truth: when you let the model write its own memory, you inherit its hallucination tendencies permanently, so a human-in-the-loop review process is not optional.
The three escape routes form a continuum of locality: trim at the tool boundary, compress within the window, and offload beyond the process. Choosing among them is a cost trade-off, not a one-size-fits-all solution.
The discussion centers on two concrete flaws in the hand-rolled agent's memory and trim logic. The memory file's append-only design stacks contradictory user preferences over time, forcing the model to weigh stale constraints against current ones with no way to tell recency. Trimming by entry count can exit early when the last entries alone exceed the budget, deferring the failure to the next request. A separate remark reinforces the need for human-readable memory and manual review, since model-generated rules drift over time.
Section 2.3 pushes external storage one step further, and you'll hit the issue that remember only appends: memory.md only grows, never shrinks. A user first says replies should be short, then half a month later says they should be detailed. Both entries are still there, and on next startup both get stuffed into the prefix, so the model oscillates between the two preferences because it can't tell which one came later. This file at least needs two things: individual entries deletable, and new entries on the same topic overwrite old ones. Otherwise, the older entries increasingly behave like hard constraints. One more thing: if trimming stops by count (e.g., stop when fewer than four entries remain), but the last two entries themselves already exceed the budget, the loop exits directly and returns something still overlong. Locally it won't surface; the error only shows up on the next request.
The point about storing memory as human-readable text is spot on — rules the model derives on its own really do go off the rails. Periodic manual checks are something you can't skip, otherwise wrong memories get more skewed the longer they're used.