# Dochplt Docmt Eval

> Evaluates document-level machine translation (DocMT) capabilities of LLMs, probing how context length, fine-tuning strategy, and multilingual training affect translation quality across diverse languages and document structures. Use when the user wants to benchmark on DocHPLT, or asks about evaluating this task. Reports BLEU.

- Skill: `qhjqhj00/dochplt-docmt-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/dochplt-docmt-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/dochplt-docmt-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/dochplt-docmt-eval

---


# dochplt-docmt-eval

> DocHPLT: A Massively Multilingual Document-Level Translation Dataset — Dayyán O'Brien et al. (2025) (arXiv:2508.13079, 2025)

## What this evaluates

Evaluates document-level machine translation (DocMT) capabilities of LLMs, probing how context length, fine-tuning strategy, and multilingual training affect translation quality across diverse languages and document structures.

## Datasets

- **DocHPLT** — total ?; splits: test (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard n-gram precision with brevity penalty. Computed by treating each hypothesis and reference document as a single string, then averaging scores across all documents. Config: nrefs:1, case:mixed, eff:no, tok:13a, smooth:exp, version:2.5.1.
- `chrF++` — range: percent
  - Character n-gram F-score. Computed by treating each hypothesis and reference document as a single string, then averaging scores across all documents. Config: nrefs:1, case:mixed, eff:yes, nc:6, nw:2, space:no, version:2.5.1.

## Input / output format

**Input**: Source document (or a chunk of up to 10 sentences) passed through the LLM's chat template as a single prompt string.

**Output**: Translated target document (or chunk) generated in a single pass, returned as a single string.

## Scoring recipe

```python
bleu_scores = []
chrf_scores = []
for doc in test_set:
    pred = model.generate(doc.source)
    # Treat entire document as single string for metric computation
    bleu_scores.append(bleu([doc.ref], pred, nrefs=1, case='mixed', eff=False, tok='13a', smooth='exp'))
    chrf_scores.append(chrf([doc.ref], pred, nrefs=1, case='mixed', eff=True, nc=6, nw=2, space=False))
avg_bleu = sum(bleu_scores) / len(bleu_scores)
avg_chrf = sum(chrf_scores) / len(chrf_scores)
```

## Common pitfalls

- Using sentence-level alignment to compute metrics, which contradicts the document-level evaluation protocol and is explicitly avoided by the authors.
- Assuming full document-to-document translation always outperforms chunked context; the paper shows 10-sentence chunks often yield better BLEU/chrF++ than full-doc training.
- Generalizing multilingual fine-tuning benefits as universally positive; performance gains are model-dependent and inconsistent across languages.

## Evidence (verbatim from paper)

> We compute BLEU555nrefs:1—case:mixed—eff:no—tok:13a—smooth:exp—version:2.5.1 (Papineni et al., [2002]) and chrF++666nrefs:1—case:mixed—eff:yes—nc:6—nw:2—space:no—version:2.5.1 (Popović, [2017]) by treating each hypothesis document and reference document as a single string, and then averaging these scores across all documents. Our metric choice avoids the need for sentence-level alignment, which DocMT outputs do not guarantee.

## Citation

```bibtex
@misc{obrien2025dochplt,
  title={DocHPLT: A Massively Multilingual Document-Level Translation Dataset},
  author={Dayyán O'Brien et al. (2025)},
  year={2025},
  note={arXiv:2508.13079}
}
```

- arXiv: 2508.13079