Multiple Choice Vqa Eval

multiple-choice-vqa-eval

Mitigating Easy Option Bias in Multiple-Choice Question Answering — Zhang et al. (2025) (arXiv:2508.13428, 2025)

What this evaluates

This evaluation probes the true multimodal reasoning capability of vision-language models on multiple-choice question answering tasks. It specifically measures whether models rely on actual question understanding or exploit visual relevance imbalances between correct answers and distractors. Performance is assessed under both standard (vision, question, options) and question-omitted (vision, options) settings to detect easy-option bias.

Datasets

• NExT-QA — total ?; splits: train (34132), val (4996)
• MMStar — total 1500; splits: eval (1500)

Metrics

• accuracy (primary) — range: percent
  • Standard multiple-choice accuracy: the proportion of correctly predicted answers out of the total number of questions. Calculated as (number of correct predictions) / (total questions).

Input / output format

Input: For each instance, the model receives a visual input (8 sampled frames for videos or a single image), a natural language question, and a set of multiple-choice options (typically 1 correct + 4 distractors).

Output: A single selected option label (e.g., A, B, C, D, or E) corresponding to the predicted answer.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Models may achieve high accuracy by matching visual features in the options to the image without reading the question, artificially inflating scores on standard benchmarks.
• Randomly sampled negative options are often visually irrelevant, making them too easy to reject and failing to test true reasoning capabilities.
• Evaluating only under the (V, Q, O) setting masks the easy-option bias; the (V, O) ablation is required to reveal whether models actually use the question.

Evidence (verbatim from paper)

We observe that: (1) GroundAttack significantly decreases accuracies across all five VLMs compared to the original negative options, when Easy-Options Bias is mitigated. For example, Qwen2.5VL-7B drops from 79.56% to 50.36%, and DeepSeek-VL2-Tiny decreases from 59.55% to 25.80%, which approaches the random guessing baseline of 20% (given 5 options: 1 positive and 4 negatives).

Citation

@misc{zhang2025mitigating,
  title={Mitigating Easy Option Bias in Multiple-Choice Question Answering},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2508.13428}
}

• arXiv: 2508.13428