Polarmem Multimodal Eval

polarmem-multimodal-eval

PolarMem: A Training-Free Polarized Latent Graph Memory for Verifiable Multimodal Agents — Chen et al. (2026) (arXiv:2602.00415, 2026)

What this evaluates

Evaluates training-free multimodal agents on retrieval-augmented generation, general reasoning, and hallucination robustness by testing a polarized latent graph memory that injects logical constraints at inference time.

Datasets

• MRAMG-Bench — total ?; splits: test (-1)
• MRAG-Bench — total ?; splits: test (-1)
• Visual-RAG — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• HallusionBench — total ?; splits: test (-1)

Metrics

• performance (primary) — range: percent
  • Standard benchmark performance score calculated as the percentage of correctly answered questions or appropriately graded responses per each benchmark's official evaluation protocol.

Input / output format

Input: Multimodal query (image + text prompt) from the benchmark, augmented with retrieved evidence snippets formatted as '[Fact Check: {Status}] {Content}', truncated to a fixed token limit per item to fit the context window.

Output: Text response/answer generated by the VLM backbone to the query.

Scoring recipe

def compute_performance(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if evaluate_answer(p, g) == 'correct')
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Residual retrieval noise in long contexts can cause diminishing or negative returns on stronger models.
• Retrieved items are truncated to a fixed maximum length (in tokens) to fit within the model context window, which may discard relevant evidence.
• All memory-based methods must use the same memory sources and the same retrieval budget (Top-K evidence) to ensure fair comparison.

Evidence (verbatim from paper)

Table 1: Performance comparison on six benchmarks. PolarMem (PolarMem) significantly boosts retrieval-augmented benchmarks (MRAMG/MRAG/Visual-RAG) for smaller and mid-size backbones, while showing diminishing or occasionally negative returns on stronger models for general reasoning (MMMU/MMStar) and hallucination robustness (HallusionBench), likely due to residual retrieval noise in long contexts.

Citation

@misc{chen2026polarmem,
  title={PolarMem: A Training-Free Polarized Latent Graph Memory for Verifiable Multimodal Agents},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2602.00415}
}

• arXiv: 2602.00415