A LongMemEval-S number you can reproduce

We held off on posting a benchmark for a long time. Not because we didn't have runs - because most memory benchmarks you read are a number with no way to check it. A blog says "X%", and you have no idea what reader answered the questions, what judge scored them, how much context the retriever was allowed to feed, or whether an LLM quietly did the hard part inside the "memory" layer. So the number tells you almost nothing about the memory system. Here is one we're comfortable standing behind, because you can run it yourself. The result On LongMemEval-S , the full 500-question set, Engrava 0.6.0 scored 81.6% micro in August 2026 - 81.76% averaged across the six question categories. The run uses the canonical LongMemEval scorer (pinned to a known upstream commit), the standard gpt-4o-2024-08-06 reader and judge over the OpenAI API, and a top_k of 20 retrieved turns. Nothing about the reader, the prompt, or the scorer is ours; the only thing we swapped in is the memory. It is compared against the previous release: 0.5.0, run in July 2026, scored 82.4% micro / 82.58% macro on the same 500 questions, same reader, same judge, same scorer, same top_k . Both rows are on the leaderboard, both verified , and both ship their reproduction artifacts. We are leading with 0.6.0 because that is the version this post is about; the older row stays because removing it when the number goes down is exactly the move that makes benchmark pages worthless. 0.5.0 (2026-07-10) 0.6.0 (2026-08-11) micro 82.4% 81.6% macro 82.58% 81.76% n 500 500 Both figures are dated on purpose. This post is a record of two specific runs, not a running scoreboard; the current table, whatever version is newest when you read this, lives on the Engrava benchmarks page . The run also has no LLM in the memory pipeline. Ingestion and retrieval are deterministic - hybrid search over a typed graph, no model doing extraction, summarization, or re-ranking behind the curtain. In the benchmark's own terms this is a Group A