Include Eval

include-eval

INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge — Romanou et al. (2024) (arXiv:2411.19799, 2024)

What this evaluates

Evaluates multilingual language understanding and regional/cultural knowledge across 44 languages using native-language exam questions. Probes models' ability to handle region-specific contexts without English bias or translation artifacts, and measures performance variance across languages and prompting strategies.

Datasets

• INCLUDE-base — total 22635; splits: test (22635)
• INCLUDE-lite — total 10770; splits: test (10770)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard exact-match accuracy for multiple-choice questions: the proportion of instances where the model's predicted option exactly matches the gold answer.

Input / output format

Input: Multiple-choice question with four answer options, presented in either the target language or English. Optionally includes a regional prefix prompt to contextualize culture/region. Few-shot context (5 examples) or zero-shot (with CoT suffix) may be prepended.

Output: Model must select one of the four answer options.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return correct / len(golds)

# predictions and golds are lists of option indices/labels
# zero-shot CoT outputs are parsed to extract the final option
# 5-shot and zero-shot scores are reported separately

Common pitfalls

• Evaluation cost is high; e.g., GPT-4o with 5-shot demonstrations exceeded $1000, making full-scale testing prohibitive for many researchers.
• Model language support often does not cover all 44 benchmark languages, leading to uneven evaluation coverage and potential data contamination or leakage in training sets.
• Prompt language (In-Language vs. English) and regional prefix significantly impact scores, so results are not directly comparable across prompting settings without explicit reporting.

Evidence (verbatim from paper)

For standardization (and alignment with prior benchmarks; Hendrycks et al., [2020]), Include-base and Include-lite contain only multiple-choice questions with four answer options. Questions from Include with fewer than four options were omitted during sampling, and questions with more than four options were pruned of options until only four remained. Following Hendrycks et al. ([2020]), we report both 5-shot and zero-shot scores.

Citation

@misc{romanou2024include,
  title={INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge},
  author={Romanou et al. (2024)},
  year={2024},
  note={arXiv:2411.19799}
}

• arXiv: 2411.19799