Wmt2024 Discourse Literary Eval

wmt2024-discourse-literary-eval

Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translation — Wang et al. (2024) (arXiv:2412.11732, 2024)

What this evaluates

Evaluates document-level and literary-domain machine translation quality, focusing on discourse-aware metrics like consistency, anaphora resolution, and term consistency across long-form Chinese web novels. The protocol combines automated n-gram and neural metrics with a structured human judgment framework to capture cross-sentence coherence and literary style.

Datasets

• WMT 2024 Discourse-Level Literary Translation Shared Task — total ?; splits: test (-1); repo https://github.com/longyuewangdcu/GuoFeng-Webnovel

Metrics

• d-BLEU (primary) — range: [0, 100]
  • Document-level sacreBLEU calculated by concatenating all sentences in a document into a single line and computing n-gram matches case-sensitively using sacrebleu tool with signature nrefs:2|case:mixed|eff:no|tok:13a|smooth:exp|version:2.3.1.
• Human Quality Score — range: [0, 5]
  • 0–5 scale assessing general quality (fluency, adequacy) and discourse-aware quality (consistency, word choice, anaphora). Scores are averaged across two professional evaluators.

Input / output format

Input: Source Chinese document (concatenated sentences for automated metrics) or window of neighboring sentences with document context for human evaluation.

Output: Translated target language document (German, Russian, or English).

Scoring recipe

def score_d_bleu(hypothesis_sentences, reference_sentences):
    doc_hyp = ' '.join(hypothesis_sentences)
    doc_ref = ' '.join(reference_sentences)
    return sacrebleu.corpus_bleu(doc_hyp, [doc_ref], tokenize='13a', case_mixed=True).score

def score_human(hyp_text, ref_text, context):
    # Evaluators assign 0-5 based on rubric for general & discourse quality
    score_1 = evaluate(hyp_text, ref_text, context, evaluator=1)
    score_2 = evaluate(hyp_text, ref_text, context, evaluator=2)
    return (score_1 + score_2) / 2

Common pitfalls

• d-BLEU requires concatenating all sentences into a single line before scoring, unlike standard sentence-level BLEU.
• Human evaluation uses a 0-5 scale for two distinct dimensions (general quality and discourse-aware quality), not a single MQM score.
• Automatic metrics show significant discrepancies with human judgments, especially for discourse properties like anaphora and consistency.

Evidence (verbatim from paper)

For document-level evaluation, we employ document-level sacreBLEU (d-BLEU) Liu et al. ([2020]), which assesses n-gram matches across complete documents. This involves concatenating all sentences in a document into a single line before applying the sacreBLEU metric. Evaluations are conducted in a case-sensitive manner. We establish two sets of evaluation criteria: 1) general quality, covering aspects such as fluency and adequacy; 2) discourse-aware quality, including factors such as consistency, word choice, and anaphora. The detailed scoring criteria are listed in Table[6]. Accordingly, each output will be assigned two distinct scores (0∼5).

Citation

@misc{wang2024wmt2024discourse,
  title={Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translation},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2412.11732}
}

• arXiv: 2412.11732