Discox Eval

discox-eval

DiscoX: Benchmarking Discourse-Level Translation task in Expert Domains — Zhao et al. (2025) (arXiv:2511.10984, 2025)

What this evaluates

Evaluates machine translation systems on discourse-level coherence and terminological precision in expert domains. It probes the model's ability to maintain long-form text consistency and handle domain-specific language beyond sentence-level translation.

Datasets

• DiscoX — total 200; splits: test (200); repo https://github.com/ByteDance-Seed/DiscoX

Metrics

• Metric-S (primary) — range: [0, 100]
  • A reference-free, LLM-as-a-judge scoring system that evaluates translations across three modular dimensions: accuracy, fluency, and appropriateness. Scores are aggregated into an overall score normalized to a [0, 100] scale.

Input / output format

Input: Source text (Chinese or English) and its machine-translated target text.

Output: Numerical scores for Accuracy, Fluency, Appropriateness, and an Overall score (0-100 scale).

Scoring recipe

def evaluate(source, translation):
    acc = llm_judge(source, translation, dim='accuracy')
    flu = llm_judge(source, translation, dim='fluency')
    app = llm_judge(source, translation, dim='appropriateness')
    overall = aggregate(acc, flu, app)
    return {'accuracy': acc, 'fluency': flu, 'appropriateness': app, 'overall': overall}

Common pitfalls

• Metric-S is reference-free; do not provide gold translations to the judge model.
• Validation uses pairwise ranking consistency (SPA), not absolute score correlation like Pearson or Spearman.
• Segment-level ties are handled by treating scores as consistent if their difference is less than 0.05.

Evidence (verbatim from paper)

The most striking finding is in the overall average consistency: Metric-S achieves 70.3%, more than doubling the 34.7% score of XCOMET-QE, a SOTA baseline.

Citation

@misc{zhao2025discox,
  title={DiscoX: Benchmarking Discourse-Level Translation task in Expert Domains},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2511.10984}
}

• arXiv: 2511.10984