Mirage Score

mirage-score

Mirage The Illusion of Visual Understanding — Asadi et al. (2026) (arXiv:2603.21687, 2026)

What this evaluates

This metric probes whether multimodal models rely on genuine visual grounding or exploit textual priors and benchmark structures to answer questions without actual image input. It quantifies the mirage effect where models generate confident, visually descriptive answers or achieve high accuracy despite the complete absence of visual data.

Datasets

• VQA-RAD — total 430; splits: test (430)
• MedXpertQA-MM — total 2000; splits: test (2000)
• MicroVQA — total 1042; splits: test (1042)
• MMMU-Pro — total 1730; splits: test (1730)
• Video-MME — total 2700; splits: test (2700)
• Video-MMMU — total 900; splits: test (900)

Metrics

• mirage-score (primary) — range: percent
  • Mirage Score = (Accuracy in mirage-mode / Accuracy in original mode) × 100%. It measures the ratio of a model's benchmark accuracy when visual inputs are omitted versus when they are provided.

Input / output format

Input: Per instance: question text, answer options (if multiple-choice), and optionally an image or video frames. In mirage-mode evaluation, all visual inputs are stripped but the prompt template remains identical to the original condition.

Output: Multiple-choice: exactly one letter formatted as [[X]]. Yes/No: 'Yes' or 'No'. Open-ended: free-text response. Final answer must be enclosed in double brackets.

Scoring recipe

acc_original = sum(1 for p, g in zip(preds_original, golds) if p == g) / len(golds)
acc_mirage = sum(1 for p, g in zip(preds_mirage, golds) if p == g) / len(golds)
if acc_original == 0:
    return 0.0
mirage_score = (acc_mirage / acc_original) * 100
return mirage_score

Common pitfalls

• Models may output uncertainty or refuse to answer when images are missing, artificially deflating the mirage score if not handled consistently across conditions.
• Prompt templates must remain strictly identical between original and mirage conditions; any added text acknowledging missing images shifts the task to explicit guessing rather than measuring the mirage effect.
• Reasoning mode parameters (e.g., temperature, reasoning_effort) vary across APIs and can significantly alter accuracy in both modes, requiring careful parameter matching to avoid confounding results.

Evidence (verbatim from paper)

We define the mirage score for a given model–benchmark pair as: Mirage Score = (Accuracy in mirage-mode / Accuracy in original mode) × 100%. The per-model mirage score is the average mirage score across all six benchmarks, quantifying the model’s overall dependence on the mirage effect.

Citation

@misc{asadi2026mirage,
  title={Mirage The Illusion of Visual Understanding},
  author={Asadi et al. (2026)},
  year={2026},
  note={arXiv:2603.21687}
}

• arXiv: 2603.21687