Mqud Eval

mqud-eval

Multimodal QUD: Inquisitive Questions from Scientific Figures — Wu et al. (2026) (arXiv:2604.23733, 2026)

What this evaluates

This benchmark evaluates whether vision-language models can generate scientifically grounded, figure-dependent questions rather than generic visual queries. It probes content-specific visual grounding by measuring how model outputs change when the correct figure is replaced, removed, or kept, alongside assessing the depth and diversity of the generated questions.

Datasets

• MQUD — total 251; splits: test (200), analysis (51)

Metrics

• rIG (primary) — range: other
  • Relative Information Gain. Computed as the normalized reduction in question negative log-likelihood (NLL) when conditioning on the correct figure versus text-only. Higher values indicate stronger figure dependency.
• swap_gap — range: other
  • Δ^swap_F = L_swap(Q_F) - L_to(Q_F), where L_swap is the NLL of the question generated with a wrong figure from the same paper, and L_to is the NLL with no figure. Positive values indicate content-specific grounding.
• swap_pos_rate — range: percent
  • Percentage of evaluation items where swap_gap > 0, indicating the model relies on specific figure content rather than generic visual cues.
• question_quality — range: percent
  • LLM-judged preference rates across dimensions (depth, figure specificity, diversity) via blind A/B testing against human expert ratings.

Input / output format

Input: Title, abstract, figure image, and caption (trigger context). Surrounding paragraphs are provided for answer generation but questions are triggered only by the trigger context.

Output: A single natural language question ($Q_F$) grounded in the provided figure and text.

Scoring recipe

def compute_metrics(predictions, gold, model):
    nll_correct = [model.nll(q, fig) for q, fig in zip(predictions, gold['figures'])]
    nll_none = [model.nll(q, None) for q in predictions]
    nll_swap = [model.nll(q, gold['swap_figures']) for q in predictions]
    rIG = [(n - c) / n for n, c in zip(nll_none, nll_correct)]
    swap_gap = [s - n for s, n in zip(nll_swap, nll_none)]
    swap_pos_rate = sum(1 for g in swap_gap if g > 0) / len(swap_gap) * 100
    quality = llm_judge.evaluate(predictions, gold['questions'])
    return rIG, swap_gap, swap_pos_rate, quality

Common pitfalls

• Confusing generic visual bias (where any figure lowers loss) with true content-specific grounding (where a wrong figure increases loss relative to no figure).
• Assuming text-only fine-tuning achieves equivalent visual grounding; ablation shows text-only SFT yields low rIG despite similar swap rates, proving the metrics capture distinct properties.
• Relying solely on proprietary models for evaluation, which prevents controlled diagnostics like training-dynamics tracking and swap-based ablations.

Evidence (verbatim from paper)

rIG increases from 0.60 [0.49, 0.73] to 0.97 [0.71, 1.25] ($p<0.0001$; Table6.1), and per-type analysis shows that figure-driven types retain more visual dependence than integration types (AppendixB). Notably, a text-only SFT ablation achieves comparable swap (76%) but much lower rIG (0.27), confirming that the two diagnostics capture distinct properties: swap reflects discourse-structural patterns, while rIG requires genuine visual grounding (Table7).

Citation

@misc{wu2026multimodalqud,
  title={Multimodal QUD: Inquisitive Questions from Scientific Figures},
  author={Wu et al. (2026)},
  year={2026},
  note={arXiv:2604.23733}
}

• arXiv: 2604.23733