Scientific Figure Mcqa Eval

scientific-figure-mcqa-eval

When Choices Become Priors: Contrastive Decoding for Scientific Figure Multiple-Choice QA — Roh et al. (2026) (arXiv:2603.28026, 2026)

What this evaluates

Evaluates a model's ability to perform high-level visual reasoning and domain-specific knowledge grounding on scientific figures within a multiple-choice question answering setting. It specifically probes whether models can resist choice-induced prior bias where text-only answer options incorrectly steer predictions away from visually supported ground truth.

Datasets

• MAC — total ?; splits: test (-1)
• SciFIBench — total ?; splits: test (-1)
• MMSci — total 3711; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1] or percent
  • Fraction of correctly predicted answer options out of the total number of instances.
• Macro-F1 — range: [0, 1] or percent
  • Harmonic mean of precision and recall computed per answer class, then averaged across all classes to mitigate label imbalance.

Input / output format

Input: A scientific figure (image) paired with a multiple-choice question and a fixed set of candidate answer options. Some benchmarks also include accompanying textual descriptions or context.

Output: A single selected answer option (e.g., A, B, C, or D) corresponding to the model's highest-scoring candidate.

Scoring recipe

def compute_accuracy(preds, gold):
    return sum(1 for p, g in zip(preds, gold) if p == g) / len(gold)

def compute_macro_f1(preds, gold, classes):
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
    return sum(f1s) / len(f1s)

Common pitfalls

• MMSci exhibits a highly imbalanced answer-label distribution and variable candidate counts, causing Accuracy and Macro-F1 to diverge when gains are concentrated on frequent labels.
• Contrastive decoding baselines (VCD, ICD) and the proposed method rely on hyperparameter coefficients (e.g., α=0.5) that weight the original vs. contrastive branches; default settings are used without per-dataset tuning.
• Models are prone to choice-induced prior bias, where semantically plausible text-only options are selected over visually correct ones, potentially inflating performance if not explicitly measured.

Evidence (verbatim from paper)

Performance is evaluated using Accuracy and Macro-F1 to account for potential class imbalance across benchmarks. We provide the formal definition of Macro-F1 in Appendix[A.1]. For VCD and ICD, we adopt the default hyperparameter settings from the original implementations without additional tuning. For SciCon, we use $\alpha=0.5$ as the default setting throughout the main experiments.

Citation

@misc{roh2026whenchoices,
  title={When Choices Become Priors: Contrastive Decoding for Scientific Figure Multiple-Choice QA},
  author={Roh et al. (2026)},
  year={2026},
  note={arXiv:2603.28026}
}

• arXiv: 2603.28026