OmniMemEval Puts 14 Memory Systems Through the Same Gauntlet
Memory benchmarks have been a mess of incompatible configurations, making it impossible to know whether a system actually helps an agent or just burns tokens. OmniMemEval gives teams a single, reproducible yardstick that measures both accuracy and the context cost required to achieve it—the two numbers that determine whether a memory layer is viable in production.
The OmniMemEval framework runs 14 memory systems—Mem0, Zep, Letta, Supermemory, Hindsight, and others—through the same models, prompts, and judge criteria to produce directly comparable scores. It evaluates on two axes: Agent Memory (SWE-Bench, BrowseComp, LiveCodeBench, OmniMath, GDPVal) and User Memory (LoCoMo, LongMemEval, PersonaMem-v2, HaluMem, BEAM), using both OpenClaw and Hermes agents.
MemOS, the framework's reference system, posts the highest overall scores on LoCoMo (88.83), LongMemEval (89.20), and PersonaMem-v2 (40.58) while using far fewer context tokens than competitors. On Agent Memory tasks, it lifts OpenClaw's GDPVal accuracy from 34.48% to 62.07% and Hermes' SWE-Bench accuracy from 37.18% to 52.56%.
The evaluation also tracks cost: average turns and output characters per task. MemOS achieves its scores with shorter outputs and fewer interaction rounds, meaning the gains come from better memory retrieval and context organization rather than from stuffing more tokens into the prompt.
Most memory system vendors publish scores using different underlying models, prompts, and retrieval settings, which makes their numbers non-comparable. OmniMemEval's insistence on a single judge model and identical answer-model configuration across all systems is the first serious attempt to fix this.
The context-token numbers expose a dirty secret: several systems achieve passable accuracy only by shoving enormous context windows into the prompt. MemOS's 4,151 tokens versus graphiti-zep's 117,106 on LongMemEval is not a small gap—it is the difference between a system that retrieves and one that dumps.
PersonaMem-v2's focus on sensitive preferences, safety boundaries, and forgetting requests tests something most memory evals ignore: whether the system respects what it should not remember. A high score here with low context tokens suggests the retrieval is precise enough to avoid leaking sensitive information into unrelated queries.
The Evolve pipeline's shift from full-trajectory processing to DAG-based span batching is a cost play that matters more than the accuracy numbers. Running reflection and compression on every step of a 50-step task is economically non-viable; processing only the high-information-density spans makes continuous agent learning affordable.