Wili 2018 Eval

wili-2018-eval

The WiLI benchmark dataset for written language identification — Thoma (2018) (arXiv:1801.07779, 2018)

What this evaluates

Evaluates the ability of models to correctly identify the language of monolingual text paragraphs. It probes language identification capabilities across a wide range of languages (235) with balanced representation.

Datasets

• WiLI-2018 — total 1000; splits: test (1000)

Metrics

• F1 (primary) — range: [0, 1]
  • Standard classification F1 score, calculated per language and averaged (typically micro-averaged for the overall benchmark score). Precision = TP/(TP+FP), Recall = TP/(TP+FN), F1 = 2PR/(P+R).

Input / output format

Input: A monolingual text paragraph (approximately 1,000 tokens).

Output: A single ISO 639-3 language code (or Wikipedia-specific code for some languages).

Scoring recipe

def compute_f1(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(predictions)

Common pitfalls

• The dataset uses some non-standard or Wikipedia-specific language codes (e.g., 'map-bms', 'lzh') that may not match standard ISO 639-3 lists.
• Text is pre-normalized to Unicode NFKC form, so models must handle normalization or the benchmark assumes it.
• Results are reported per-language; overall performance should be micro-averaged across all 235 languages.

Evidence (verbatim from paper)

Lang Prec Recall F1 Lang Prec Recall F1

Citation

@misc{thoma2018wili,
  title={The WiLI benchmark dataset for written language identification},
  author={Thoma (2018)},
  year={2018},
  note={arXiv:1801.07779}
}

• arXiv: 1801.07779