Multimodalqa Eval

multimodalqa-eval

MultiModalQA: Complex Question Answering over Text, Tables and Images — Talmor et al. (2021) (arXiv:2104.06039, 2021)

What this evaluates

Evaluates complex question answering capabilities that require joint reasoning across text, tables, and images. It probes multi-hop reasoning, cross-modal inference, and the ability to align and process structured and unstructured data to produce correct answer lists.

Datasets

• MultiModalQA — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Average F1 score computed over lists of gold and predicted answers, aligning them as described in Dua et al. (2019).
• Exact Match (EM) — range: [0, 1]
  • Exact match accuracy between gold and predicted answer lists.

Input / output format

Input: A question requiring reasoning over provided text, tables, and/or images.

Output: A list of answer strings.

Scoring recipe

def compute_metrics(predictions, golds):
    f1_scores = []
    em_scores = []
    for pred, gold in zip(predictions, golds):
        # Align gold and predicted lists per Dua et al. (2019)
        f1_scores.append(calculate_token_f1(pred, gold))
        em_scores.append(1.0 if set(pred) == set(gold) else 0.0)
    return sum(f1_scores) / len(f1_scores), sum(em_scores) / len(em_scores)

Common pitfalls

• Automatic evaluation is non-trivial due to the need to align lists of answers across modalities.
• Human performance can be affected by context length and fatigue, leading to errors not present in models.
• Approximately 8% of questions contain weak distractors or redundant evidence, making them easier than intended.

Evidence (verbatim from paper)

Our evaluation metrics need to support lists of answers, and thus we use average F1 and Exact Match (EM), as described in Dua et al. (2019), where answers on the gold and predicted lists are aligned.

Citation

@misc{talmor2021multimodalqa,
  title={MultiModalQA: Complex Question Answering over Text, Tables and Images},
  author={Talmor et al. (2021)},
  year={2021},
  note={arXiv:2104.06039}
}

• arXiv: 2104.06039