Deepl Supertext Comparison Eval

deepl-supertext-comparison-eval

A comparison of translation performance between DeepL and Supertext — Flückiger et al. (2025) (arXiv:2502.02577, 2025)

What this evaluates

This evaluation probes the translation quality and contextual consistency of two commercial machine translation systems (DeepL and Supertext) by having professional raters perform blind pairwise comparisons on full documents. It specifically measures whether LLM-based long-context translation yields superior document-level coherence compared to traditional segment-level systems.

Datasets

• Unspecified source documents — total ?; splits: test (-1); repo https://github.com/supertext/evaluation_deepl_supertext

Metrics

• pairwise preference rate (primary) — range: [0, 1]
  • Calculated as the proportion of segments where a given system's translation is chosen as 'better' out of all non-equal judgments. Equal judgments are excluded from the denominator to compute a strict win rate.

Input / output format

Input: Full source document with segments in original order, including formatting such as newlines. Side-by-side translations labeled Translation A and Translation B for each segment.

Output: Per segment, a categorical choice: 'Translation A is better', 'Translation B is better', or 'equal quality'.

Scoring recipe

def compute_preference_rate(wins_a, wins_b, ties):
    decisive = wins_a + wins_b
    if decisive == 0:
        return 0.0
    rate_a = wins_a / decisive
    rate_b = wins_b / decisive
    return rate_a, rate_b

Common pitfalls

• Evaluating segments in isolation rather than using the full document context as instructed.
• Allowing punctuation or formatting differences to bias the preference judgment instead of focusing on content.
• Misinterpreting the 'equal' option as a tie-breaker rather than a valid third category that should be excluded in win-rate calculations.

Evidence (verbatim from paper)

For each segment in each document, the assigned rater is asked to choose whether Translation A is better, Translation B is better, or whether both translations are of equal quality.

Citation

@misc{fluckiger2025deeplsupertext,
  title={A comparison of translation performance between DeepL and Supertext},
  author={Flückiger et al. (2025)},
  year={2025},
  note={arXiv:2502.02577}
}

• arXiv: 2502.02577