Exams QA Eval

exams-qa-eval

EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answering — Hardalov et al. (2020) (arXiv:2011.03080, 2020)

What this evaluates

Evaluates multilingual and cross-lingual question answering capabilities on high school-level exams across multiple subjects and languages. Probes domain-specific reasoning, knowledge retrieval, and zero-shot transfer between languages with varying linguistic and subject overlaps.

Datasets

• EXAMS — total 24000; splits: test (-1); repo http://github.com/mhardalov/exams-qa

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of questions) * 100.

Input / output format

Input: A multiple-choice question with four options (A-D), optionally accompanied by retrieved Wikipedia context paragraphs.

Output: A single predicted option label (A, B, C, or D) corresponding to the correct answer.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
accuracy = (correct / len(gold_labels)) * 100

Common pitfalls

• Assuming simple information retrieval (IR) over Wikipedia is sufficient; the dataset requires complex domain-specific reasoning beyond word matching.
• Overlooking subject-level performance variations; models struggle significantly with Chemistry and Physics compared to other subjects.
• Ignoring the impact of subject overlap and language family similarity on cross-lingual zero-shot transfer performance.

Evidence (verbatim from paper)

In Table 4, we show the overall per-language performance of the evaluated models. ... IR is better than random guessing, but it is clear that most questions require reasoning beyond simple word matching. ... We can see that the Natural Science questions are the most challenging ones, which is mostly due to Chemistry and Physics.

Citation

@misc{hardalov2020exams,
  title={EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answering},
  author={Hardalov et al. (2020)},
  year={2020},
  note={arXiv:2011.03080}
}

• arXiv: 2011.03080