Madlad 400 Mt Eval

madlad-400-mt-eval

MADLAD-400: A Multilingual And Document-Level Large Audited Dataset — Kudugunta et al. (2023) (arXiv:2309.04662, 2023)

What this evaluates

Evaluates the multilingual machine translation and zero-shot/few-shot translation capabilities of models trained on the MADLAD-400 dataset. It probes cross-lingual generalization, low-resource language handling, and the impact of data auditing on translation quality across multiple benchmarks and language pairs.

Datasets

• WMT — total ?; splits: test (-1)
• Flores-200 — total ?; splits: test (-1)
• NTREX — total ?; splits: test (-1)
• GATONES — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram overlap metric computed via sacreBLEU with sentence-level averaging and tokenization.
• chrF — range: percent
  • Character-level F-score computed via sacreBLEU, measuring n-gram overlap at the character level.

Input / output format

Input: For MT models: source sentence with a target language token (e.g., <2xx>) prepended. For LM models: few-shot prompt [s1]: X1\n[t1]: Y1\n... [s1]: X\n[t1]:

Output: Target sentence string.

Scoring recipe

import sacrebleu
def evaluate(predictions, references):
    bleu_score = sacrebleu.corpus_bleu(predictions, [references]).score
    chrf_score = sacrebleu.corpus_chrf(predictions, [references]).score
    return {'bleu': bleu_score, 'chrf': chrf_score}

Common pitfalls

• Using different tokenization or BLEU versions than sacreBLEU yields non-comparable scores.
• Few-shot LM evaluation randomly samples 1000 examples from NTREX as a dev set for demonstrations, causing run-to-run variance.
• Flores-200 evaluation uses a computationally constrained 272-pair subset, not the full dataset.

Evidence (verbatim from paper)

We use the sacreBLEU [55] implementation of bleu and chrF as metrics. We evaluate our trained models on the following datasets: WMT. We use the 15 WMT languages frequently used to evaluate multilingual machine translation models by Siddhant et al. [61], Kim et al. [38], Kudugunta et al. [42], NLLBTeam et al. [51]: cs, de, es, fi, fr, gu, hi, kk, lv, lt, ro, rs, es, tr and zh. Flores-200. We evaluate on the languages in the Flores-200 dataset [51] that overlap with the languages available in either MADLAD-400 or the parallel data described in Section 3. NTREX. We evaluate on the languages in the recently introduced NTREX dataset [23]. Gatones. Finally, we evaluate on the languages in GATONES, the in-house, 38-language eval set used in [9] and the GATitos paper [36].

Citation

@misc{kudugunta2023madlad400,
  title={MADLAD-400: A Multilingual And Document-Level Large Audited Dataset},
  author={Kudugunta et al. (2023)},
  year={2023},
  note={arXiv:2309.04662}
}

• arXiv: 2309.04662