# Madlad 400 Mt Eval

> Evaluates the multilingual machine translation and zero-shot/few-shot translation capabilities of models trained on the MADLAD-400 dataset. It probes cross-lingual generalization, low-resource language handling, and the impact of data auditing on translation quality across multiple benchmarks and language pairs. Use when the user wants to benchmark on WMT, Flores-200, NTREX, GATONES, or asks about evaluating this task. Reports BLEU.

- Skill: `qhjqhj00/madlad-400-mt-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/madlad-400-mt-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/madlad-400-mt-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/madlad-400-mt-eval

---


# madlad-400-mt-eval

> MADLAD-400: A Multilingual And Document-Level Large Audited Dataset — Kudugunta et al. (2023) (arXiv:2309.04662, 2023)

## What this evaluates

Evaluates the multilingual machine translation and zero-shot/few-shot translation capabilities of models trained on the MADLAD-400 dataset. It probes cross-lingual generalization, low-resource language handling, and the impact of data auditing on translation quality across multiple benchmarks and language pairs.

## Datasets

- **WMT** — total ?; splits: test (-1)
- **Flores-200** — total ?; splits: test (-1)
- **NTREX** — total ?; splits: test (-1)
- **GATONES** — total ?; splits: test (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard n-gram overlap metric computed via sacreBLEU with sentence-level averaging and tokenization.
- `chrF` — range: percent
  - Character-level F-score computed via sacreBLEU, measuring n-gram overlap at the character level.

## Input / output format

**Input**: For MT models: source sentence with a target language token (e.g., <2xx>) prepended. For LM models: few-shot prompt [s1]: X1\n[t1]: Y1\n... [s1]: X\n[t1]:

**Output**: Target sentence string.

## Scoring recipe

```python
import sacrebleu
def evaluate(predictions, references):
    bleu_score = sacrebleu.corpus_bleu(predictions, [references]).score
    chrf_score = sacrebleu.corpus_chrf(predictions, [references]).score
    return {'bleu': bleu_score, 'chrf': chrf_score}
```

## Common pitfalls

- Using different tokenization or BLEU versions than sacreBLEU yields non-comparable scores.
- Few-shot LM evaluation randomly samples 1000 examples from NTREX as a dev set for demonstrations, causing run-to-run variance.
- Flores-200 evaluation uses a computationally constrained 272-pair subset, not the full dataset.

## Evidence (verbatim from paper)

> We use the sacreBLEU [55] implementation of bleu and chrF as metrics. We evaluate our trained models on the following datasets: WMT. We use the 15 WMT languages frequently used to evaluate multilingual machine translation models by Siddhant et al. [61], Kim et al. [38], Kudugunta et al. [42], NLLBTeam et al. [51]: cs, de, es, fi, fr, gu, hi, kk, lv, lt, ro, rs, es, tr and zh. Flores-200. We evaluate on the languages in the Flores-200 dataset [51] that overlap with the languages available in either MADLAD-400 or the parallel data described in Section 3. NTREX. We evaluate on the languages in the recently introduced NTREX dataset [23]. Gatones. Finally, we evaluate on the languages in GATONES, the in-house, 38-language eval set used in [9] and the GATitos paper [36].

## Citation

```bibtex
@misc{kudugunta2023madlad400,
  title={MADLAD-400: A Multilingual And Document-Level Large Audited Dataset},
  author={Kudugunta et al. (2023)},
  year={2023},
  note={arXiv:2309.04662}
}
```

- arXiv: 2309.04662