Memorybench Eval

memorybench-eval

MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems — Ai et al. (2025) (arXiv:2510.17281, 2025)

What this evaluates

This benchmark evaluates how well LLM-based systems retain and utilize both declarative and procedural memory across diverse domains and task formats. It specifically probes continual learning capabilities by measuring performance improvements when systems process explicit and implicit user feedback over multiple interaction sessions.

Datasets

• MemoryBench (Domain & Task Format Partitions) — total ?; splits: train (-1), test (-1); repo https://github.com/LittleDinoC/MemoryBench

Metrics

• LLM-as-Judge score (primary) — range: [0, 1] (normalized)
  • Each dataset's original metrics are merged into a single 1-10 score using an LLM-as-Judge. Results are then min-max normalized or z-scored within each dataset, and averaged across all test cases in a partition to yield the final score.

Input / output format

Input: Initial task context (corpus) and query, optionally augmented with historical feedback logs (dialog sessions, action feedback like 'like'/'copy') from previous training cases.

Output: Text response to the query.

Scoring recipe

# 1. Score each test case with LLM-as-Judge (1-10)
raw_scores = [llm_judge(response, gold) for response, gold in test_cases]
# 2. Normalize within each dataset (min-max or z-score)
if use_minmax:
    norm_scores = (raw_scores - min(raw_scores)) / (max(raw_scores) - min(raw_scores))
else:
    mean, std = np.mean(raw_scores), np.std(raw_scores)
    norm_scores = (raw_scores - mean) / std
# 3. Average across datasets in the partition
final_score = np.mean(norm_scores)

Common pitfalls

• Context length limits cause systems like Mem0 to fail or truncate on long-input tasks (LiSo, LiLo), artificially lowering scores if not accounted for.
• Mixing historical feedback logs with current task context introduces noise, as many systems fail to differentiate between the two.
• Off-policy evaluation (pre-generated feedback) yields different results than on-policy (interactive feedback), and many memory systems are too slow for on-policy runs.

Evidence (verbatim from paper)

MemoryBench follows the original evaluation metrics of each dataset. However, because certain datasets (JuDGE, IdeaBench, and SciTechNews) have multiple evaluation criteria in their official settings, we simplify the process by employing a LLM-as-Judge paradigm to merge these metrics into a single ranging from 1 to 10. We present the prompts used for this aggregation in [A.1]. For the overall evaluation of a task or domain, we apply min-max normalization or compute the z score to all results within each dataset before calculating the average performance of all test cases.

Citation

@misc{ai2025memorybench,
  title={MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems},
  author={Ai et al. (2025)},
  year={2025},
  note={arXiv:2510.17281}
}

• arXiv: 2510.17281