Hplt V2 Eval

hplt-v2-eval

An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT) — Burchell et al. (2025) (arXiv:2503.10267, 2025)

What this evaluates

Evaluates the quality of the HPLT v2 multilingual corpus by training downstream models (masked language models, generative LMs, and MT systems) and measuring their performance on standard linguistic, natural language understanding, and machine translation benchmarks.

Datasets

• Universal Dependencies (UD) treebanks — total ?; splits: test (-1); repo https://github.com/hplt-project/HPLT-WP4
• WikiAnn — total ?; splits: test (-1)
• FLORES-200 — total ?; splits: test (-1)

Metrics

• CoNLL 2018 F1/accuracy — range: [0, 1]
  • Standard CoNLL 2018 evaluation script computes token-level accuracy and label-based F1 for POS, lemmatization, and dependency parsing.
• seqeval balanced F1 — range: [0, 1]
  • Sequence evaluation metric computing strict BIO-format F1, averaged across entity types.
• BLEU (primary) — range: [0, 1]
  • n-gram precision with brevity penalty, computed via sacrebleu (nrefs:1|case:mixed|eff:no|smooth:exp|version:2.5.1).
• chrF++ — range: [0, 1]
  • Character n-gram F-score with nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1.
• COMET-22-DA — range: [0, 1]
  • Neural reference/DA metric scoring translation quality on a 0-1 scale.

Input / output format

Input: Pretraining corpora (HPLT v2) for model training; benchmark datasets (UD, WikiAnn, FLORES-200) for downstream evaluation.

Output: Model predictions (part-of-speech tags, dependency parses, entity labels, or translated text) compared against gold standard annotations.

Scoring recipe

def compute_metrics(predictions, references, gold_tags=None, gold_entities=None):
    # MT metrics
    bleu = sacrebleu.corpus_bleu(predictions, [references], nrefs=1, case='mixed', smooth='exp', version='2.5.1')
    chrf = sacrebleu.corpus_chrf(predictions, [references], nrefs=1, case='mixed', eff=True, nc=6, nw=0, space=False)
    comet = comet_model.evaluate(predictions, references)
    # UD/NER metrics
    conll_f1 = conll2018_eval.evaluate(gold_tags, predictions)
    seqeval_f1 = seqeval.evaluate(gold_entities, predictions, mode='strict', scheme='bio')
    # Average only over directions/tasks covered by all compared models
    valid_scores = [s for s in [bleu, chrf, comet, conll_f1, seqeval_f1] if s is not None]
    return mean(valid_scores)

Common pitfalls

• Win rate is calculated as the count of languages where a model outperforms others, not an average score difference.
• BLEU and COMET averages are computed only over translation directions covered by all models in a given comparison, excluding missing directions.
• Lemmatization performance differences <1% are considered statistically non-significant.

Evidence (verbatim from paper)

We evaluate all models on the FLORES-200 benchmark (NLLB Team et al., [2024]) using BLEU (Papineni et al., [2002]), chrF++ (Popović, [2017]), and COMET-22-DA (Rei et al., [2022]). We use sacrebleu’s implementation of BLEU272727nrefs:1|case:mixed|eff:no|smooth:exp|version:2.5.1, and where applicable, tok:ja-mecab, tok:ko-mecab, or tok:13a and chrF++282828nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1 with signatures footnoted (Post, [2018]).

Citation

@misc{burchell2025hplt,
  title={An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)},
  author={Burchell et al. (2025)},
  year={2025},
  note={arXiv:2503.10267}
}

• arXiv: 2503.10267