Learning Long-Term Memory from the Hermes Agent: The Core Is Not a Vector Database, but Memory Governance

The core problem that Agent long-term memory needs to solve is:

After a task ends, which experiences should be retained? How should these experiences influence behavior in the next task?

Many long-term memory solutions start with RAG or vector databases, but vector databases mainly solve the problem of "similar content recall." A complete long-term memory system also needs to handle writing, revision, forgetting, promotion, evidence tracking, and task context construction.

Therefore, long-term memory is not just "historical record retrieval," but a governance closed loop centered around experience accumulation, evidence management, rule promotion, and context construction.

1. Typical Problem: Why Does an Agent Repeatedly Step into the Same Trap?

Suppose there is a travel planning Agent.

The first time, a user asks to plan a three-day family trip to Beijing. The Agent's initial draft schedules the Forbidden City and Universal Studios on the same day. The route check fails because these two locations are far apart across districts, and it's unrealistic to travel back and forth in an afternoon with a 5-year-old child.

Subsequently, the Agent revises the plan: concentrate attractions in the same area, and allocate a full separate day for distant, large-scale projects like Universal Studios. The user also adds an extra instruction: "For future itineraries like this, also attach the route check conclusion."

A week later, the user asks to plan a two-day trip to Xi'an. The Agent again schedules the Terracotta Warriors and the Muslim Quarter on the same day. The Terracotta Warriors are far from the city center, essentially the same type of planning error as the previous Beijing case.

The problem exposed here is not a lack of single-reasoning capability, but that the failure evidence, correction strategy, and user preference from the last task were not accumulated as long-term memory. After the session ends, the context is lost. When the next task begins, the Agent does not bring back these experiences and naturally repeats the mistake.

2. RAG Does Not Equal Long-Term Memory

RAG mainly solves retrieval, which is the recall problem:

Given a batch of materials, find the segments relevant to the current problem.

But long-term memory involves at least four actions:

Action	Problem Solved
Ingestion	What experiences should enter the memory system
Revision	How old memories are merged, rewritten, updated
Forgetting	What content should no longer influence behavior
Retrieval	Which memories should be fetched for a new task

RAG mainly covers the last item.

The key difficulty of long-term memory lies in the first three items: which information is worth writing, how to turn it into stable experience, and when it should no longer enter the context.

The boundary between the two can be understood as:

Storage and retrieval solve "can it be read back," while memory governance solves "should this record influence future behavior."

3. A Four-Layer Memory Model

A relatively clear long-term memory structure can be broken down into four layers:

Layer	Role	Example
Raw Episodic	Raw event stream	User input, assistant reply, tool call results
Consolidated Episodic	Session summaries and candidate experiences	Where a task failed, how it was later corrected
Semantic / Policy	Stable rules and long-term preferences	Distant large projects occupy a full day alone; user requires attaching check conclusions
Working Memory	Temporary context for the current task	Rules and summaries that truly need to be brought into the prompt for this task

Each layer has different responsibilities:

• The Raw layer is only responsible for preserving original evidence, without semantic processing.
• The Consolidated layer is responsible for compressing a session into summaries and candidate experiences.
• The Semantic / Policy layer only retains a small number of high-confidence rules.
• Working Memory is dynamically generated at the start of a new task and is not permanent storage.

This means long-term memory should not simply stuff all history back into the prompt, but should first govern the history into usable experiences, then recall them by task.

4. Why Are Raw Events Not Enough?

If only raw conversations are saved and then simply retrieved, the recall results are usually still fragments.

For example, a new task is: "Help me plan a two-day family trip to Shanghai. The child wants to go to Disneyland, and the final route must be executable."

The system might recall these historical fragments:

• Beijing route check failed: Forbidden City to Universal Studios is too far.
• Xi'an route check failed: Terracotta Warriors are too far from the city center.
• Corrected route check passed.
• The user said to attach check conclusions in the future.

Although these fragments are relevant, they are still a stream of events. The truly reusable experience is:

Distant, large-scale projects should occupy a full day alone and should not be mixed with city-center attractions.

This experience usually does not exist in a single raw event, but is extracted from failures, corrections, tool passes, and user feedback.

Therefore, Raw events are only a source of evidence and cannot be directly equated with long-term memory. Long-term memory requires further extraction, compression, and promotion.

5. From Candidate Memories to Stable Rules

Long-term memory can be generated in two steps.

First, use a model to extract candidate memories from a session, for example:

• The user is traveling with a 5-year-old child who needs a nap from 2 PM to 4 PM.
• The user does not like influencer-style check-in itineraries; a day should not be too packed.
• When outputting an itinerary, attach the route check conclusion.
• Distant, large-scale projects cannot be squeezed into half a day.

Each candidate memory should carry evidence, such as the original event ID. Memories without evidence cannot be verified or rolled back later.

Second, not all candidates directly become long-term rules; they must go through a promotion judgment.

Common promotion conditions can include:

Promotion Channel	Meaning
Cross-session recurrence	Similar experience verified across multiple tasks
Tool failure evidence	For example, a route check returns "failed"
User explicit long-term instruction	User expresses "do it this way from now on"

This step is suitable for a "model + code" division of labor:

• The model is responsible for natural language understanding, such as judging whether two candidates are synonymous.
• Code is responsible for structured judgment, such as whether the tool status is "failed," whether it recurs across sessions, or whether there is a "durable" marker.

A more prudent principle is:

Extraction is done by the model, governance is constrained by rules and evidence.

6. The Semantic Layer Must Be Restrained

The Semantic / Policy layer is the layer with the widest influence in long-term memory.

Rules in this layer will enter future tasks with higher priority. If written incorrectly or over-generalized, subsequent tasks will be affected.

For example, "stay near Wangfujing" only appeared in one Beijing trip and without the user explicitly stating it is long-term, it is not suitable for promotion to a stable rule. It can be kept in the session summary but should not influence future tasks as a long-term preference.

Content more suitable for promotion includes:

• The user explicitly stated that future itinerary outputs should attach route check conclusions.
• Tools have repeatedly proven that distant, large-scale projects are not suitable for half-day mixing.
• The user explicitly expressed that the child needs a nap and the itinerary should not be too packed.

Therefore, rules in the Semantic layer should follow:

Few in number, high confidence, evidence-backed, traceable.

7. Heartbeat Tidying: Let Memory Self-Evolve in the Background

Long-term memory tidying is not suitable for real-time execution in every dialogue round.

When a user is waiting for a reply, if summarization, extraction, and merging are additionally performed in every round, it will increase latency. A more reasonable approach is a background heartbeat mechanism:

1. Periodically scan new Raw events.
2. Find unprocessed events.
3. Trigger summary extraction, candidate generation, and rule promotion.
4. Write a checkpoint to record processing progress.
5. Adjust the next heartbeat interval based on the number of new events.

This keeps the dialogue process lightweight and moves memory tidying to system idle time.

8. The Risk of Heartbeats: Automatic Tidying Can Become Automatic Pollution

The content read by the background heartbeat is not always trustworthy. Emails, web pages, chat records, repository issues, etc., can all enter the event stream.

If background memory writing lacks governance, misinformation, rumors, or misleading content could be written into long-term memory and influence behavior in subsequent tasks.

Therefore, background writing should meet the following conditions:

• Has a source.
• Has evidence.
• Has a promotion reason.
• Has a scope boundary.
• Is rollback-able.
• Does not bypass governance policies.

Without these constraints, automatic tidying can evolve into automatic pollution.

9. When a New Task Starts, Don't Re-inject All History

When a new task arrives, the correct approach is not to stuff all historical sessions back into the prompt, but to construct a task-specific brief, i.e., Working Memory.

It typically contains:

• Current task description.
• Project scope.
• Relevant stable rules.
• Relevant session summaries.
• Evidence sources.

For example, for a new Shanghai family trip task, Working Memory can bring in:

• The child needs rest from 2 PM to 4 PM.
• The itinerary cannot be too packed.
• Important attractions should be scheduled for the morning if possible.
• Distant, large-scale projects like Disneyland should occupy a full day alone.
• The output result must attach the route check conclusion.

This way, the Agent receives a curated task context, not a pile of raw historical streams.

10. Scope Isolation Is Important

Long-term memory also needs to handle boundaries between different projects, users, and scenarios.

For example, a user says in an expense reimbursement scenario:

From now on, small-amount transportation tickets should be pre-audited directly according to the new travel policy.

This rule should belong to the reimbursement Agent and should not be recalled by the travel planning Agent.

Therefore, each memory should carry a scope, such as:

• project_id
• user_id
• session_id
• domain
• source

Long-term memory without a scope can easily become a source of global pollution.

11. The Position of the Vector Database

A vector database is suitable as an acceleration component for synonym recall, but not as the sole source of truth.

A more reasonable positioning is:

The vector database is responsible for semantic recall, while the main facts remain in traceable structured files or databases.

Reasons include:

• Vector databases are not good at expressing complete evidence chains.
• Vector databases are not suitable as the sole basis for permission and scope governance.
• Recall indexes can be rebuilt, but original evidence cannot be lost.
• Stable rules need to be auditable and rollback-able.

A more prudent architecture is:

• JSONL or databases store Raw, Summary, Rule, and Evidence.
• The vector database stores rebuildable indexes.
• Recall results must be traceable back to the original evidence.

12. Final Acceptance Must Look at Behavioral Differences

Whether long-term memory is effective cannot just be judged by "whether content was recalled."

The real acceptance test should see whether the output behavior for the same task undergoes explainable and verifiable changes with and without memory.

For example, for the same Shanghai family trip task, with memory it should reflect:

• Disneyland occupies a full day alone.
• A 14:00-16:00 nap window is explicitly reserved.
• The itinerary is not too packed.
• Key arrangements are placed in the morning.
• The route check conclusion is attached in the output.
• It can explain how long-term rules were implemented.

If the system only recalls the rules but the final output does not comply with them, then the long-term memory has not truly taken effect.

Therefore, the focus of memory system acceptance should be:

Not the recall list, but whether the task result produces an explainable behavioral change.

Summary

Agent long-term memory can be summarized as a governance closed loop:

1. Raw conversations and tool results enter the Raw layer.
2. After a session ends, episode summaries and candidate experiences are extracted.
3. Candidate memories are promoted to stable rules based on evidence.
4. A background heartbeat periodically triggers tidying and checkpointing.
5. When a new task starts, Working Memory is generated.
6. Finally, verify whether the memory is effective through behavioral differences.

The core conclusion is:

The goal of long-term memory is not to remember more, but to only let evidence-backed, traceable, and governed memories influence future behavior.

This is also the most noteworthy aspect of the long-term memory design similar to the Hermes Agent: it does not build memory as an infinitely growing historical warehouse, but breaks it down into stable layers, historical layers, tidying layers, and recall layers, allowing the Agent to continuously accumulate experience from past failures, corrections, and user preferences.