qhjqhj00 mem-gallery-eval
Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents — Bei et al. (2026) (arXiv:2601.03515, 2026)
What this evaluates
This benchmark evaluates multimodal long-term conversational memory in MLLM agents across multi-session dialogues. It probes the agent's ability to extract, adapt, reason over, and manage evolving visual and textual information, including handling temporal dependencies, conflicting updates, and knowledge gaps.
Datasets
- Mem-Gallery — total 1711; splits: test (1711); repo https://github.com/YuanchenBei/Mem-Gallery
Metrics
answer correctness(primary) — range: [0, 1]- Percentage of QA pairs where the model's generated answer exactly matches the ground-truth answer. Evaluated per subtask (e.g., Factual Retrieval, Knowledge Resolution) and aggregated across the benchmark.
Input / output format
Input: A sequence of multi-session multimodal conversations (text and images) with temporal gaps, followed by a question requiring the model to retrieve or reason over accumulated memory.
Output: A textual answer to the question.
Scoring recipe
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, gold_answers):
if pred.strip().lower() == gold.strip().lower():
correct += 1
return correct / total
Common pitfalls
- Models often fail to persist visual information across sessions without explicit captioning or retrieval mechanisms.
- Evaluating 'answer refusal' requires checking if the model correctly abstains when information is missing or contradictory, rather than just penalizing wrong answers.
- Long-horizon conversations require incremental memory updates; models relying solely on context windows will fail.
Evidence (verbatim from paper)
LLMs are first used to check answer correctness and question clarity. This verification step is followed by careful human revision.
Citation
@misc{bei2026memgallery,
title={Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents},
author={Bei et al. (2026)},
year={2026},
note={arXiv:2601.03515}
}
- arXiv: 2601.03515