Mia Eval

mia-eval

Memory Intelligence Agent — Jingyang Qiao et al. (2026) (arXiv:2604.04503, 2026)

What this evaluates

Evaluates the reasoning, tool-use, and memory-augmented planning capabilities of agents on complex multi-hop QA and visual question answering tasks. It probes how well models leverage episodic memory, test-time learning, and reflection to improve answer accuracy over iterative search trajectories.

Datasets

• FVQA-test — total ?; splits: test (-1)
• InfoSeek — total ?; splits: test (-1)
• MMSearch — total ?; splits: test (-1)
• SimpleVQA — total ?; splits: test (-1)
• LiveVQA — total ?; splits: test (-1)
• In-house 1 — total ?; splits: test (-1)
• In-house 2 — total ?; splits: test (-1)
• HotpotQA — total ?; splits: test (-1)
• 2WikiMultiHopQA — total ?; splits: test (-1)
• SimpleQA — total ?; splits: test (-1)
• GAIA (text-only subset) — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of questions answered correctly, determined by an external LLM judge (Qwen3-32B) comparing the model's final output to the ground truth label.

Input / output format

Input: Natural language questions, optionally accompanied by images, along with access to external search tools (wiki25 for text-to-text search, Serper for image-to-image and text-to-text search).

Output: A short, precise final answer string generated after multi-turn tool calling and reasoning.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        # Qwen3-32B LLM Judger evaluates correctness per Appendix 13
        is_correct = llm_judge_check(pred, gold['answer'])
        if is_correct:
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Long-context memory methods (e.g., RAG, Mem0) often underperform simpler baselines due to noise introduced by extended contexts.
• Correctness is determined entirely by an external LLM judge (Qwen3-32B), so results depend heavily on the judge's prompt template and model version.
• Tool usage is unconstrained, leading to high variance in reasoning paths and call counts across different models and runs.

Evidence (verbatim from paper)

MIA achieves the highest overall performance among open-source models, reaching an average accuracy of 53.6. Compared to the previous best memory-based method, MIA improves the average accuracy by 5.5, including specific increases of 3.3 on FVQA-test, 6.4 on the multi-hop task LiveVQA, and an impressive 9.1 on the highly challenging custom task In-house 1.

Citation

@misc{qiao2026memoryintelligenceagent,
  title={Memory Intelligence Agent},
  author={Jingyang Qiao et al. (2026)},
  year={2026},
  note={arXiv:2604.04503}
}

• arXiv: 2604.04503