qhjqhj00 segale-doc-eval
Extending Automatic Machine Translation Evaluation to Book-Length Documents — Wang et al. (2025) (arXiv:2509.17249, 2025)
What this evaluates
Evaluates whether a document-level machine translation evaluation framework can robustly handle translation anomalies (over-translation, under-translation, boundary shifts) and effectively score long-form texts without predefined sentence boundaries.
Datasets
- SEGALE Test Set — total ?; splits: original (-1), over-translate (-1), under-translate (-1), flex-boundary (-1); repo https://github.com/nvlabs/SEGALE
Metrics
correlation with human judgments(primary) — range: [-1, 1]- Pearson or Spearman correlation between the averaged segment-level document scores and human MQM annotations. Human judgments are averaged at the segment level to form document-level scores.
NA ratio— range: percent- Proportion of null alignments in the predicted segmentation/alignment. The paper reports the absolute deviation from the ground truth NA ratio (|Δ_Gold|) as an auxiliary measure.
Input / output format
Input: Source document, system translation document, and optionally a reference document. Ground truth sentence boundaries and alignments are provided for baseline comparisons.
Output: Document-level translation score (aggregated from segment-level scores) and null alignment ratio.
Scoring recipe
# 1. Segment & align source/target documents
segments_src, segments_tgt = segale_segment_and_align(src_doc, tgt_doc)
# 2. Compute segment-level scores using underlying metric (e.g., COMET)
seg_scores = [underlying_metric(s, t) for s, t in zip(segments_src, segments_tgt)]
# 3. Average to get document-level score
doc_score = mean(seg_scores)
# 4. Compute correlation with human MQM judgments
corr = pearsonr(doc_scores, human_mqm_scores)
# 5. Compute NA ratio deviation
na_ratio = count(null_alignments) / total_segments
na_deviation = abs(na_ratio - ground_truth_na_ratio)
Common pitfalls
- Interpreting a 0% NA ratio deviation as proof of a good evaluation scheme; the paper explicitly warns that perfect NA ratio alone does not guarantee accurate quality assessment.
- Assuming scores higher than the 'Gold' baseline indicate a better method; the authors note this is likely an artifact of segmentation variations and not a meaningful improvement.
- Confusing the underlying MT quality metrics (COMET, MetricX) with the meta-evaluation metric (correlation with human judgments).
Evidence (verbatim from paper)
Similar to previous work and preliminary experiments, we use correlation between document-level scores and human judgments as the primary metric. ... We also report NA ratio for each method as the auxiliary metric. Ideally, we would like to achieve the same NA ratio as the groundtruth (|Δ_Gold|=0), but the reader should note that perfect NA ratio on its own doesn’t necessarily imply a good evaluation scheme.
Citation
@misc{wang2025segale,
title={Extending Automatic Machine Translation Evaluation to Book-Length Documents},
author={Wang et al. (2025)},
year={2025},
note={arXiv:2509.17249}
}
- arXiv: 2509.17249