Wmt2023 Discourse Literary Eval

wmt2023-discourse-literary-eval

Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMs — Wang et al. (2023) (arXiv:2311.03127, 2023)

What this evaluates

Evaluates machine translation systems on discourse-level literary translation from Chinese to English, focusing on long-range context, cultural adaptation, and stylistic fidelity across entire web novels rather than isolated sentences.

Datasets

• WMT 2023 Discourse-Level Literary Translation Test Set — total ?; splits: test (-1)

Metrics

• d-BLEU (primary) — range: [0, 100]
  • Document-level BLEU calculated by concatenating all continuous sentences in a single book into one line, then scoring the concatenated hypothesis against the concatenated reference using sacreBLEU.
• Human MQM Score — range: [0, 100]
  • Literary-adapted Multidimensional Quality Metrics (MQM) framework scoring translation quality across Accuracy, Fluency, Style, Terminology, Localization, and Other error types, averaged across four annotators.

Input / output format

Input: Chinese source text from web novels (provided as sentence-level segments for automatic metrics or full documents for d-BLEU).

Output: English translated text corresponding to the source segment/document.

Scoring recipe

# d-BLEU
doc_hyp = " ".join(hypothesis_sentences)
doc_ref = " ".join(reference_sentences)
score = sacreBLEU.corpus_score(doc_hyp, [doc_ref]).score
return score

# Human MQM
scores = [annotator.evaluate(doc_hyp, doc_ref) for annotator in annotators]
return sum(scores) / len(scores)

Common pitfalls

• Automatic metrics like BLEU and COMET fail to capture literary style, cultural nuance, and discourse coherence, leading to poor correlation with human judgment.
• d-BLEU requires concatenating entire books into single lines, which differs from standard sentence-level evaluation and may penalize formatting or paragraph breaks.
• Human evaluation relies on a small sampled subset (5 documents, ~2,194 words) for ranking, which may not fully represent system performance on the broader test set.

Evidence (verbatim from paper)

The evaluation metrics includes 1) sentence-level BLEU,chrF,COMET, TER; and 2) document-level d-BLEU. To calculate d-BLEU, we first concatenate all continuous sentences in one book as on line, and then employ sacreBLEU to obtain scorers.

Citation

@misc{wang2023wmt2023discourse,
  title={Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMs},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2311.03127}
}

• arXiv: 2311.03127