Mkqa Eval

mkqa-eval

MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering — Longpre et al. (2020) (arXiv:2007.15207, 2020)

What this evaluates

Evaluates multilingual open-domain question answering models on their ability to generate or extract short, factoid answers across 26 typologically diverse languages. It specifically probes cross-lingual transfer, handling of unanswerable queries, and robustness to language-specific normalization and threshold tuning for abstention.

Datasets

• MKQA — total 260000; splits: test (260000); repo https://github.com/apple/ml-mkqa

Metrics

• token overlap F1 (primary) — range: [0, 1]
  • Token-level F1 score computed after normalizing both prediction and gold answer by removing whitespace, punctuation, and articles. The final reported score is the Macro Average F1, calculated as the mean of per-language F1 scores.

Input / output format

Input: A question q^l in a target language l.

Output: A prediction p^l consisting of either a 'Text Answer' (sequence of tokens in language l) or 'No Answer', along with a continuous 'No Answer probability' score used for threshold tuning.

Scoring recipe

def normalize(text):
    return remove_whitespace_punctuation_articles(text)

def compute_example_f1(pred, gold):
    p_norm = normalize(pred)
    g_norm = normalize(gold)
    if p_norm == g_norm: return 1.0
    p_tokens = set(p_norm.split())
    g_tokens = set(g_norm.split())
    if not g_tokens: return 0.0
    prec = len(p_tokens & g_tokens) / len(p_tokens)
    rec = len(p_tokens & g_tokens) / len(g_tokens)
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0

best_lang_f1 = 0
for threshold in range(0, 101):
    preds = [p if prob <= threshold else 'No Answer' for p, prob in predictions]
    f1s = [compute_example_f1(p, g) for p, g in zip(preds, golds)]
    lang_f1 = sum(f1s) / len(f1s)
    if lang_f1 > best_lang_f1: best_lang_f1 = lang_f1

macro_f1 = sum(best_lang_f1 for each language) / num_languages
return macro_f1

Common pitfalls

• Forgetting to apply language-specific normalization (removing whitespace, punctuation, and articles) before computing F1.
• Not tuning the 'No Answer' probability threshold per language; the protocol requires reporting the best F1 over the full threshold range to eliminate threshold-tuning bias.
• Treating 'Long Answers' as distinct from 'Unanswerable'; MKQA groups long answers into the unanswerable category, so predicting 'No Answer' for them is valid and scored accordingly.

Evidence (verbatim from paper)

Every prediction $p^{l}_{i}$ is scored based on exact match (EM) and token overlap F1, as with previous open-retrieval QA datasets. The official evaluation script also ingests a “No Answer probability” for each example. If the probability is above a chosen threshold value then the prediction defaults to No Answer instead of the provided Textual Answer. ... We follow NQ in reporting the best F1 over the range of thresholds, to remove threshold tuning as a factor in evaluation. A best threshold is computed and applied per language, where each example receives a “textual” (token overlap) F1 after language-specific normalization (removing whitespace, punctuation, and articles) is applied to both the prediction and gold answers. Finally, the official per-language F1 is computed as the mean of example F1s, and the official Macro Average F1 is the mean of per-language F1 scores.

Citation

@misc{longpre2020mkqa,
  title={MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering},
  author={Longpre et al. (2020)},
  year={2020},
  note={arXiv:2007.15207}
}

• arXiv: 2007.15207