Wmt24 Chat Translation Eval

wmt24-chat-translation-eval

Findings of the WMT 2024 Shared Task on Chat Translation — Mohammed et al. (2024) (arXiv:2410.11624, 2024)

What this evaluates

Evaluates machine translation systems for bilingual customer support conversations, focusing on context utilization, discourse coherence, and turn-level versus conversation-level translation quality across five language pairs.

Datasets

• MAIA 2.0 — total ?; splits: train (79400), dev (12300), test (10000)

Metrics

• COMET (primary) — range: [-1, 1]
  • Neural reference-based metric trained on human judgments to predict translation quality.
• BLEU — range: [0, 1]
  • Standard n-gram overlap metric computed with SacreBLEU.
• chrF — range: [0, 1]
  • Character n-gram F-score computed with SacreBLEU.
• ContextCometQE — range: [-1, 1]
  • Reference-free metric that uses bilingual context (previous two turns) to assess translation quality.
• MuDA F1 — range: [0, 1]
  • F1 accuracy on four context-dependent discourse phenomena tags: lexical cohesion, formality, pronoun resolution, and verb forms.
• DA+SQM — range: [0, 100]
  • Human assessment using Direct Assessment and Scalar Quality Metric on a continuous scale from 0 to 100 based on accuracy and grammatical correctness.
• ContextMQM — range: other
  • LLM-based fine-grained error score aggregating minor, major, and critical errors weighted by 1, 5, and 10 respectively.

Input / output format

Input: Source text in the customer's or agent's language, accompanied by preceding conversation context (full context for human eval, up to 8 turns for LLM eval, or 2 turns for ContextCometQE).

Output: Translated text in the target language (agent's or customer's language).

Scoring recipe

def compute_metrics(predictions, gold, sources):
    comet_scores = comet_model.predict(hypotheses=predictions, references=gold, sources=sources)
    comet_val = comet_scores['scores'].mean()
    bleu_val = sacrebleu.corpus_bleu(predictions, [gold]).score / 100
    chrf_val = sacrebleu.corpus_chrf(predictions, [gold]).score / 100
    muda_f1 = f1_score(tag_discourse(predictions), tag_discourse(gold))
    human_val = sum(linguist_scores) / len(linguist_scores)
    mqm_val = sum(minor*1 + major*5 + critical*10 for errors in llm_mqm_outputs)
    return {'COMET': comet_val, 'BLEU': bleu_val, 'chrF': chrf_val, 'MuDA_F1': muda_f1, 'DA+SQM': human_val, 'ContextMQM': mqm_val}

Common pitfalls

• Turn-level metrics (e.g., BLEU, COMET) often overestimate system capability because performance degrades significantly at the conversation level and in later turns.
• Standard MT metrics fail to capture discourse coherence; context-dependent phenomena like formality and pronoun resolution require specialized tagging or context-aware metrics.
• LLM-based ContextMQM evaluation was restricted to the en-de language pair due to budget constraints, limiting cross-lingual generalizability of fine-grained error analysis.

Evidence (verbatim from paper)

We use Comet Rei et al. ([2022]) as our primary evaluation metric for assessing translation quality of the submitted systems. Additionally, we report lexical metrics: BLEU and chrF using the SacreBLEU library Post ([2018]). We also include ContextCometQE (Agrawal et al., [2024]), a reference-free metric that uses bilingual context (previous two turns) to assess the translation quality of the current turn.

Citation

@misc{mohammed2024wmt24chattranslation,
  title={Findings of the WMT 2024 Shared Task on Chat Translation},
  author={Mohammed et al. (2024)},
  year={2024},
  note={arXiv:2410.11624}
}

• arXiv: 2410.11624