Dochplt Docmt Eval

dochplt-docmt-eval

DocHPLT: A Massively Multilingual Document-Level Translation Dataset — Dayyán O'Brien et al. (2025) (arXiv:2508.13079, 2025)

What this evaluates

Evaluates document-level machine translation (DocMT) capabilities of LLMs, probing how context length, fine-tuning strategy, and multilingual training affect translation quality across diverse languages and document structures.

Datasets

• DocHPLT — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram precision with brevity penalty. Computed by treating each hypothesis and reference document as a single string, then averaging scores across all documents. Config: nrefs:1, case:mixed, eff:no, tok:13a, smooth:exp, version:2.5.1.
• chrF++ — range: percent
  • Character n-gram F-score. Computed by treating each hypothesis and reference document as a single string, then averaging scores across all documents. Config: nrefs:1, case:mixed, eff:yes, nc:6, nw:2, space:no, version:2.5.1.

Input / output format

Input: Source document (or a chunk of up to 10 sentences) passed through the LLM's chat template as a single prompt string.

Output: Translated target document (or chunk) generated in a single pass, returned as a single string.

Scoring recipe

bleu_scores = []
chrf_scores = []
for doc in test_set:
    pred = model.generate(doc.source)
    # Treat entire document as single string for metric computation
    bleu_scores.append(bleu([doc.ref], pred, nrefs=1, case='mixed', eff=False, tok='13a', smooth='exp'))
    chrf_scores.append(chrf([doc.ref], pred, nrefs=1, case='mixed', eff=True, nc=6, nw=2, space=False))
avg_bleu = sum(bleu_scores) / len(bleu_scores)
avg_chrf = sum(chrf_scores) / len(chrf_scores)

Common pitfalls

• Using sentence-level alignment to compute metrics, which contradicts the document-level evaluation protocol and is explicitly avoided by the authors.
• Assuming full document-to-document translation always outperforms chunked context; the paper shows 10-sentence chunks often yield better BLEU/chrF++ than full-doc training.
• Generalizing multilingual fine-tuning benefits as universally positive; performance gains are model-dependent and inconsistent across languages.

Evidence (verbatim from paper)

We compute BLEU555nrefs:1—case:mixed—eff:no—tok:13a—smooth:exp—version:2.5.1 (Papineni et al., [2002]) and chrF++666nrefs:1—case:mixed—eff:yes—nc:6—nw:2—space:no—version:2.5.1 (Popović, [2017]) by treating each hypothesis document and reference document as a single string, and then averaging these scores across all documents. Our metric choice avoids the need for sentence-level alignment, which DocMT outputs do not guarantee.

Citation

@misc{obrien2025dochplt,
  title={DocHPLT: A Massively Multilingual Document-Level Translation Dataset},
  author={Dayyán O'Brien et al. (2025)},
  year={2025},
  note={arXiv:2508.13079}
}

• arXiv: 2508.13079