跪拜 Guibai
← Back to the summary

OmniMemEval Puts 14 Memory Systems Through the Same Gauntlet

From OpenClaw sparking a nationwide "lobster-raising" craze to agents like Hermes taking on more complex tasks, agents are moving toward executing and delivering increasingly long-running tasks.

The longer the task, the more tools invoked, and the more complex the execution chain, the more Memory becomes a critical infrastructure for agents.

However, Memory systems still lack a unified horizontal evaluation standard. Different solutions often present results based on different models, prompts, retrieval strategies, context lengths, and evaluation criteria. A single demo or a single leaderboard can hardly reflect their actual performance in long-running agent tasks.

Based on this, we have released the OmniMemEval unified evaluation framework, which incorporates 14 mainstream Memory solutions, two agent environments (OpenClaw and Hermes), and 10 Memory Benchmarks into a single evaluation system. It systematically compares Memory systems from two perspectives: agent task execution capability and long-term user memory capability.

The goal of this evaluation is not just to answer who "remembers more," but who can truly translate memory into agent task performance and continuous evolution capability.

Highlights Preview

How We Evaluate: Making Different Memory Systems Take the Same Exam

The core challenge of Memory evaluation is that the models, prompts, retrieval parameters, context lengths, and judge criteria behind different products' public results are often inconsistent. This affects result reproducibility and weakens horizontal comparability between different Memory systems.

To judge whether a Memory system is truly usable, one cannot just look at a single demo or a single leaderboard. The idea behind the OmniMemEval unified evaluation framework is to compare different products, different agent environments, and different benchmarks under the same criteria.

We compare MemOS with mainstream Memory systems, covering four dimensions:

  1. Most Comprehensive Product Coverage: Covers 14 mainstream Memory systems, including representative solutions like Mem0, Zep, Letta, Supermemory, Hindsight, EverOS, Memori, and mem9, for horizontal comparison under a unified configuration.
  2. More Comprehensive Agent Environment Coverage: Selects two mainstream agent environments, OpenClaw and Hermes, to verify the general improvement capability of Memory systems across different agent architectures.
  3. Most Comprehensive Benchmark Coverage: Covers two major directions, Agent Memory and User Memory, with a total of 10 Benchmarks, for comprehensive evaluation from task execution, tool invocation, and state retention to long-term user information understanding.
  4. More Comprehensive Evaluation Metrics: Measures not only whether Memory can improve task completion rates but also whether this improvement relies on more call turns, longer outputs, or larger contexts, thus comprehensively judging its effectiveness and cost in real-world scenarios.

Two Main Evaluation Tracks: Making Agents Perform Better and Understand Users Better

  1. Agent Memory (Making every execution the starting point for the next task): Evaluates the experience accumulation, evolution, and strategy reuse capabilities of memory systems in complex tasks, including five datasets: SWE-Bench, BrowseComp, LiveCodeBench, OmniMath, and GDPVal.

  2. User Memory (Making every interaction precipitate into long-term understanding of the user): Evaluates the memory system's ability to model, maintain, and update long-term user information, covering user profiles, preferences, relationship experiences, timelines, multi-turn fact retrieval, memory conflict updates, and personalized responses, including five datasets: LoCoMo, LongMemEval, PersonaMem-v2, HaluMem, and BEAM.

To ensure absolute reproducibility of results, all systems in this evaluation were run under completely identical conditions, including the same datasets, same models, same evaluation scripts, and a unified judge system. Additionally, all evaluation results specify the evaluation configuration, and the complete code and evaluation process have been open-sourced to ensure developers can reproduce the experimental results.

1. MemOS Stably Improves Task Completion Rates in OpenClaw and Hermes

To avoid contingency from a single agent or single task, this evaluation selected two mainstream agent environments, OpenClaw and Hermes, and covered five representative tasks: SWE-Bench, BrowseComp, LiveCodeBench, OmniMath, and GDPVal, forming a cross-agent, multi-task verification system.

After integrating the MemOS local plugin, both OpenClaw and Hermes Agents achieved comprehensive improvements over the baseline across all five Agent Memory evaluation dimensions, indicating that MemOS's benefits do not depend on a single agent form.

·Compared Agents in this evaluation: OpenClaw and Hermes Agent

·Agent Answer Model Configuration: qwen3.6-flash no_thinking mode

·Evaluation Judge Model: qwen3.6-flash thinking mode

In the OpenClaw environment, GDPVal showed the most significant improvement, with Acc increasing from 34.48% to 62.07%, a relative improvement of 80.0%. In long-process and complex delivery tasks, MemOS demonstrated stronger capabilities in context retention and phased conclusion reuse.

In the Hermes Agent environment, SWE-Bench showed the most significant improvement, with Acc increasing from 37.18% to 52.56%, a relative improvement of 41.4%. This indicates that MemOS is not only effective in retrieval tasks but also brings stable benefits in code understanding, evidence integration, and complex reasoning chains in real software engineering scenarios.

2. Agent Memory: 10 Sets of Results Verify Leadership

Agent Memory primarily evaluates AI's execution capability in real tasks, including complex reasoning, tool invocation, and task completion. This round of evaluation covered mainstream Memory products like Mem0, Supermemory, Memori, Hindsight, EverOS, mem9, and MemOS, each integrated into the OpenClaw and Hermes agent environments.

Since some products did not provide plugin integration methods, which would compromise evaluation fairness, only products supporting plugins were evaluated, forming 10 sets of uniformly comparable results.

This evaluation:

·Answer Model: qwen3.6-flash no_thinking

·Judge Model: qwen3.6-flash thinking

Agent Memory Five Evaluation Summary: 6 First Places out of 10 Evaluations

Acc counts the average first-pass rate after 3 consecutive independent runs of the same task.

In the OpenClaw environment, MemOS achieved an average Acc of 50.07% across five Agent Memory tasks, the best overall performance. It ranked first in BrowseComp, OmniMath, and GDPVal, tied for first in SWE-Bench, and third in LiveCode, maintaining stable completion rates in retrieval, reasoning, software engineering, and knowledge work tasks.

In the Hermes Agent environment, MemOS achieved an average Acc of 53.05% across five tasks, ranking first in the group, with first place in OmniMath and SWE-Bench, and second in LiveCode and BrowseComp. MemOS has more advantages in complex reasoning, software engineering, and coding tasks.

Overall, MemOS's leadership is not a chance high score from one agent but a stable benefit reproducible across two agent environments and five types of real tasks.

Cost Dimension: High Completion Rates Are Not Bought by Piling Up Context

Besides Accuracy, this round also focused on Avg Turns and Avg Chars (unit: k). They respectively reflect the interaction rounds and output scale required for the agent to complete a task: with similar or higher completion rates, fewer turns and shorter output mean the memory system can organize context more effectively and reduce redundant reasoning.

In the OpenClaw environment, MemOS performed stably in tasks requiring continuous context understanding like BrowseComp and GDPVal, while maintaining a low average output length. After integrating MemOS, the agent could better utilize existing memories to reduce redundant context construction and lower token consumption.

In the Hermes Agent environment, MemOS demonstrated better task completion efficiency. In the OmniMath test, MemOS achieved the highest accuracy (72.67%) while maintaining a low output scale; in the SWE-Bench test, MemOS achieved the highest accuracy (52.56%) and maintained low task interaction costs. MemOS can help agents utilize historical context and execution experience more effectively in real software engineering tasks.

3. MemOS Cloud Memory Capability Continuously Upgrades

To verify the capability improvement of the latest MemOS Cloud version in long-term memory tasks, we conducted comparative tests on different versions based on mainstream long-term memory evaluation benchmarks.

The latest MemOS Cloud version achieved leading performance in two long-term memory benchmarks, LoCoMo and LongMemEval, reaching 91.30 and 91.00 respectively.

Compared to earlier versions, the new version of MemOS Cloud further enhanced information understanding and retrieval capabilities in complex memory scenarios, covering key tasks such as basic fact recall, cross-time information association, multi-turn context understanding, and open-domain question answering, enabling AI agents to acquire, organize, and utilize long-term memory more accurately.

Evaluation Configuration: During the evaluation, it was found that in a small number of scenarios, even if the memory was recalled, a weak answer model could still be confused. Therefore, gpt-5.5 was used as the ANSWER_MODEL, and other configurations remained consistent with the evaluation framework below.

LoCoMo: Long Conversation Memory Q&A Capability Improved

Among them, Open-Domain improved from 55.21 to 76.04. These types of questions usually rely more on original text evidence and detail supplementation, and MemOS showed significantly stronger ability to return to sources to complete context when needed.

Temporal improved from 74.77 to 89.72. MemOS has developed more stable parsing and sorting capabilities for common relative time expressions in long-term memory, such as "next Wednesday," "in the last few days," and "after the last meeting."

LongMemEval: Cross-Session Long-Term Memory More Stable

SS-Asst improved from 69.64 to 100.00, and Multi-S improved from 73.68 to 85.71. The current latest version of MemOS Cloud showed the most significant improvement in two types of capabilities: assistant-side state continuation and cross-session information integration.

4. User Memory: Horizontal Evaluation of 14 Memory Solutions, Leading in Both Effectiveness and Cost

LoCoMo: Among 14 Solutions, MemOS Overall First

The LoCoMo horizontal evaluation covered 14 Memory solutions:

Mem0, Zep, Viking, Letta, Supermemory, Cognee, Memori, Hindsight, EverOS, MemMachine, mem9, MemoryLake, Backboard.io, MemOS. We displayed the Top 5.

Under a unified configuration, MemOS achieved 88.83 Overall, the highest score in this evaluation.

Evaluation Configuration:

·Memory Service Model and Answer Generation Model: gpt-4.1-mini-2025-04-14

·Evaluation Model: gpt-4o-mini-2024-07-18

·Metrics: LLM-as-a-judge Accuracy and Context Token

Long-term memory evaluation cannot only look at Accuracy; it must also look at Context Token. If a system relies on stuffing large amounts of context to get higher scores, cost, latency, and stability will become problems when it actually enters a production environment.

In LoCoMo, MemOS's average Context Token was 5,400, significantly lower than Cognee's 32,532, Hindsight's 24,683, and Mem0's 17,395.

MemOS's higher score in this round came from more effective memory extraction, retrieval, and context organization, not from longer context stacking.

LongMemEval: Among 12 Solutions, Cross-Session Capability First

In the LongMemEval horizontal evaluation, MemOS achieved 89.20 Overall, ranking first.

MemOS achieved 100.00 on three single-session information tasks: SS-User, SS-Asst, and SS-Pref, and also maintained leading levels in Temporal Reasoning, Multi-Session, and Knowledge Update.

Context Token directly affects inference cost, response latency, and context noise. MemOS averaged 4,151, lower than graphiti-zep local version's 117,106, Letta's 49,431, Hindsight's 29,755, and EverOS's 12,379.

If a high score depends on a large amount of context, online cost and latency will be continuously amplified; if the context is very short but answers are inaccurate, it cannot support real tasks.

MemOS's results in this round are closer to the balance needed in a production environment: not just high scores, but a better balance between long-term memory effectiveness and context cost.

PersonaMem-v2: Among 12 Solutions, Highest Overall in Personalization and Safety Scenarios

Under a unified configuration, MemOS achieved 40.58 Overall, the highest score in this evaluation.

PersonaMem-v2 focuses more on scenarios like personalized memory, sensitive preferences, safety preferences, and user forgetting requests. Therefore, it examines not only "accuracy of remembering" but also whether the system can make reasonable responses within privacy, safety, and preference boundaries.

If a system relies on a large amount of context to maintain persona preferences, sensitive information processing, and forgetting request capabilities, it will bring higher inference costs, longer latency, and greater risk of context pollution when actually entering a production environment.

In PersonaMem-v2, MemOS's average Context Token was 1,908, significantly lower than Supermemory's 4,473, Hindsight's 15,926, and Letta's 30,903, while still achieving the highest Overall.

MemOS's advantage in PersonaMem-v2 is not obtained by stacking longer contexts but comes from more effective personalized memory management, sensitive preference handling, forgetting mechanisms, and context organization capabilities.

Overall, MemOS simultaneously achieves the highest Overall and lower Context Token, making it more suitable for long-term personalized memory deployment in real agent scenarios.

5. Why MemOS Can Improve

This improvement mainly comes from three optimizations: extraction context enhancement, temporal understanding enhancement, and Evolve pipeline optimization.

  1. Extraction Context Enhancement: No Longer Understanding a Chunk in Isolation

In real conversations and documents, important information rarely falls neatly within the same chunk. A person's name might appear in the previous paragraph, and their preference in the next; the reason for a decision might be foreshadowed earlier, while the result is only explained later.

MemOS adds a context enhancement algorithm before memory extraction. The system first recalls strongly related context from the original text based on the current chunk's topic, entities, timeline, and semantic clues, then feeds the current segment and related context together into the extraction model.

This means the model no longer just looks at isolated segments but judges which facts, preferences, plans, and relationships are worth writing into memory within a more complete semantic window.

  1. Temporal Understanding Capability Enhancement: Making Time a Parseable Memory Attribute

Time is the most easily underestimated dimension in long-term memory.

Users rarely speak only in standard dates. More often, they say "next Wednesday," "by the end of the month," "in the last few days," "after the last meeting." If these expressions only enter the memory extraction and retrieval pipeline as plain text, the system might remember the event but cannot stably determine when it happened, whether it belongs to a certain time range, or whether recent information should be prioritized.

MemOS adds an independent temporal parsing module in the memory extraction phase, generating standardized timespecs for conversations containing relative times or time ranges. During search, the reranker performs temporal rendering on related memories based on the timespec, allowing the model to see "what this memory means on the timeline."

This optimization corresponds to the evaluation results: LoCoMo Temporal improved from 74.77 to 89.72, and LongMemEval Temp. Reas improved from 84.21 to 88.72.

3. Evolve Pipeline Optimization—Making Evolution More Cost-Effective

The core of memory evolution is to make AI no longer just "complete a task once" but to continuously learn, continuously align, and continuously grow in every real execution. In a previous article Product Update | MemOS Local Plugin 2.0: Simultaneous Support for Hermes Agent and OpenClaw Dual Agents, we redefined the agent's learning method: actions, observations, and feedback during tasks are precipitated into auditable, attributable, and reusable long-term memory assets.

It does not require retraining the large model but builds an external evolution system at the execution layer: the large model is responsible for general reasoning, and MemOS is responsible for understanding your local world. Code structures, personal preferences, historical pitfalls, and best paths can all be captured, scored, summarized in interaction after interaction, and evolved into experience and callable Skills.

More importantly, MemOS allows feedback to truly enter the learning loop: the environment tells the agent which step ran through, and the user tells the agent which approach better meets expectations. The system then calls Skills, historical trajectories, and topic cognition on demand, allowing the agent to act based on past experience.

At the same time, a memory system truly worth long-term use cannot only pursue "learning more" but must also achieve "learning cost-effectively." In this upgrade, we further made "reducing memory evolution cost" one of the core optimization directions.

MemOS no longer performs full processing of complete trajectories in the evolution path. Instead, it first lightly records Traces, then, during the import phase, based on the execution DAG, splits the trajectory into multiple Spans according to stages, dependencies, and result nodes, and then performs batch compression, reflection, and quality control.

For example:

A 50-step task was previously more like repeatedly watching the entire meeting recording from beginning to end.

Now it's more like first cutting the meeting into several key segments:

Where a tool call was made, where the state changed, where feedback appeared, where it's worth precipitating into experience.

In this way, the system processes Spans with higher information density, rather than repeatedly "close-reading" the complete trajectory.

Conclusion

Long-term memory is not simply storing history, nor is it stuffing all context back into the model. A truly usable Memory system needs to strike a balance between memory extraction, retrieval routing, temporal understanding, and cost control.

This evaluation covered 14 Memory systems, two types of agent environments, and 10 Benchmarks, systematically validating Memory systems from agent task execution to long-term user memory capabilities.

MemOS's advantage is not only reflected in a single benchmark score but in continuously improving the agent's long-term understanding, state retention, and experience reuse capabilities across different agents, different task types, and different memory scenarios.

Related Links:

OmniMemEval GitHub:

https://github.com/MemTensor/OmniMemEval

MemOS GitHub:

https://github.com/MemTensor/MemOS

Complete Evaluation Report Below

Agent Memory Evaluation:

https://github.com/MemTensor/OmniMemEval/blob/agent_mem_dev/docs/agent_memory/eval_res.md

User Memory Evaluation:

https://github.com/MemTensor/OmniMemEval/blob/agent_mem_dev/docs/benchmark-results.md