Terminology Aware Translation Eval

terminology-aware-translation-eval

It Takes Two: A Dual Stage Approach for Terminology-Aware Translation — Jaswal (2025) (arXiv:2511.07461, 2025)

What this evaluates

Evaluates a machine translation system's ability to balance overall translation quality with strict adherence to specified terminology constraints across different languages. It measures how well the model enforces lexical rules without degrading fluency or adequacy, particularly in morphologically complex languages.

Datasets

• EN-{DE,ES,RU} translation test sets — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: [0, 100]
  • Standard n-gram overlap metric between reference and hypothesis translations, computed with sentence averaging and smoothing. Reported as a percentage (0-100).
• chrF2++ — range: [0, 100]
  • Character-level n-gram F-score combining unigrams and bigrams with character n-grams, designed to be robust to morphological variations and spelling differences.
• terminology success rate (proper) — range: [0, 1]
  • Proportion of required terminology terms correctly inserted into the generated translation.
• terminology success rate (random) — range: [0, 1]
  • Proportion of randomly assigned terminology terms successfully inserted into the translation, used as a control to measure the model's constraint enforcement capability independent of contextual appropriateness.

Input / output format

Input: Source sentence in English

Output: Translated sentence in the target language (German, Spanish, or Russian)

Scoring recipe

def compute_metrics(hypotheses, references, required_terms):
    bleu = sacrebleu.corpus_bleu(hypotheses, [references]).score
    chrf = sacrebleu.corpus_chrf(hypotheses, [references]).score
    
    proper_sr = 0.0
    random_sr = 0.0
    for hyp, terms in zip(hypotheses, required_terms):
        if len(terms) == 0: continue
        proper_sr += sum(1 for t in terms if t in hyp) / len(terms)
        random_sr += sum(1 for t in terms if t in hyp) / len(terms)
    proper_sr /= len(hypotheses)
    random_sr /= len(hypotheses)
    return {'BLEU': bleu, 'chrF2++': chrf, 'proper_SR': proper_sr, 'random_SR': random_sr}

Common pitfalls

• High terminology success rates do not guarantee high BLEU or chrF2++ scores if constraints are forced inappropriately, compromising contextual fluency.
• Morphologically complex languages (e.g., Russian) exhibit larger performance gaps between constrained and unconstrained translation, making evaluation more sensitive to constraint handling.
• Random SR measures raw insertion capability rather than contextual appropriateness, potentially masking translation quality issues when constraints are misapplied.

Evidence (verbatim from paper)

We evaluate the system using three complementary metrics used by the WMT organizers: BLEU for overall translation adequacy, chrF2++ for character-level fluency and robustness, and terminology success rates (proper and random) to directly measure constraint satisfaction (papineni-etal-2002-bleu; popovic-2015-chrf).

Citation

@misc{jaswal2025dual,
  title={It Takes Two: A Dual Stage Approach for Terminology-Aware Translation},
  author={Jaswal (2025)},
  year={2025},
  note={arXiv:2511.07461}
}

• arXiv: 2511.07461