Agent Long-Term Memory Is a Governance Problem, Not a Retrieval Problem
Most agent memory efforts stop at vector search over chat history, which turns into a junk drawer of raw events that agents cannot reliably act on. A governance-first design — write, revise, forget, promote, scope — is what separates a memory system that actually changes agent behavior from one that just fills the prompt with noise.
A travel-planning agent that repeats the same scheduling mistake across sessions exposes the real gap: raw conversation logs and RAG-based retrieval are not long-term memory. Without a system to extract, revise, and promote experiences into stable rules, each new task starts from scratch.
The architecture that fixes this splits memory into four layers: raw episodic events, consolidated session summaries, semantic policy rules, and a dynamically built working memory. Candidate experiences are promoted to stable rules only when backed by evidence — cross-session recurrence, tool failures, or explicit user directives — and a background heartbeat handles tidying outside the request path.
Scope isolation, evidence traceability, and rollback capability prevent automatic memory from becoming automatic pollution. The final acceptance test is not whether a rule was recalled, but whether the agent’s output behavior changes in an explainable way when memory is present.
Framing long-term memory as a governance pipeline rather than a storage problem shifts the engineering focus from “how do we store more” to “what deserves to influence future decisions.”
The promotion step — moving a candidate memory to a stable rule only when backed by evidence — is the linchpin. Without it, every transient preference risks becoming permanent policy.
Background heartbeat processing is a double-edged sword: it keeps the interaction loop fast but creates a new attack surface where untrusted content can silently poison long-term memory.
Scope isolation is under-discussed in agent memory. A rule learned in a reimbursement context leaking into a travel planner is a failure mode that pure vector similarity cannot prevent.
The behavioral-difference acceptance test is a practical alternative to precision/recall metrics that don’t capture whether memory actually changes what the agent does.