qhjqhj00 murgat-attribution-eval
Multimodal Fact-Level Attribution for Verifiable Reasoning — Wan et al. (2026) (arXiv:2602.11509, 2026)
What this evaluates
Evaluates multimodal large language models' ability to generate verifiable, fact-level citations grounded in video and audio inputs. It probes whether models can correctly decompose reasoning into atomic claims and align them with precise temporal and modality-specific evidence without hallucinating references.
Datasets
- Video-MMMU — total 100; splits: test (100)
- WorldSense — total 100; splits: test (100)
Metrics
answer accuracy— range: [0, 100]- String matching between the model's predicted answer choice and the golden answer choice.
MURGAT-S(primary) — range: [0, 100]- Automated score combining coverage (fraction of claims with citations) and attribution quality (precision/recall of temporal and modality alignment). Normalized to 0-100. Correlates with human judgments at r=0.84.
Input / output format
Input: Multimodal prompts containing video and/or audio inputs paired with questions. Models are evaluated under three prompting variants: direct generation (BASE), generation with inline citations (+CITATION), and post-hoc sentence-level citation generation (POST-HOC).
Output: Free-form reasoning text, a final answer choice, and optionally timestamped/modality-tagged citations for each claim or sentence.
Scoring recipe
def score_accuracy(pred, gold):
return 1.0 if pred == gold else 0.0
def score_murgat_s(claims, citations):
coverage = len(citations) / max(len(claims), 1)
attribution = compute_alignment_precision_recall(claims, citations)
return (coverage * attribution) * 100
Common pitfalls
- High answer accuracy does not guarantee valid grounding; models frequently hallucinate citations while answering correctly.
- Post-hoc attribution methods may force-align abstract reasoning steps to random video segments, inflating coverage but destroying attribution quality.
- Vision-only models can generate audio citations despite lacking audio encoders, leading to misleadingly high grounding scores.
Evidence (verbatim from paper)
Overall, models struggle significantly with multimodal attribution, achieving a peak MURGAT-S of 69.2 on WorldSense and 56.9 on Video-MMMU (Gemini-3-Flash).
Citation
@misc{wan2026multimodal,
title={Multimodal Fact-Level Attribution for Verifiable Reasoning},
author={Wan et al. (2026)},
year={2026},
note={arXiv:2602.11509}
}
- arXiv: 2602.11509