Codet Eval

codet-eval

CODET: A Benchmark for Contrastive Dialectal Evaluation of Machine Translation — Alam et al. (2023) (arXiv:2305.17267, 2023)

What this evaluates

This benchmark probes the robustness of machine translation systems to dialectal variations by measuring how consistently they translate semantically similar sentences in standard vs. dialectal forms. It evaluates whether models maintain translation quality and coherence when exposed to lexical and morphosyntactic variations across multiple languages.

Datasets

• CODET — total ?; splits: test (-1); repo https://github.com/mahfuzibnalam/dialect_mt

Metrics

• COMET (primary) — range: [0, 1]
  • A neural framework that leverages both the source input and a target-language reference translation to predict MT quality, correlating with human judgments. In this protocol, the standard-variety output serves as the pseudo-reference.
• BLEU — range: [0, 100]
  • Compares n-gram matches between the candidate translation and the reference translation to determine similarity. Calculated using SacreBLEU.

Input / output format

Input: A pair of sentences: a dialectal input sentence ($\mathbf{x}$) and its standard-variety counterpart ($\tilde{\mathbf{x}}$).

Output: Two translations: $\mathbf{y}$ (model output for $\mathbf{x}$) and $\tilde{\mathbf{y}}$ (model output for $\tilde{\mathbf{x}}$).

Scoring recipe

scores = []
for x, x_tilde in dataset:
    y = model.translate(x)
    y_tilde = model.translate(x_tilde)
    # Use standard output as pseudo-reference
    score = comet(candidate=y, reference=y_tilde, source=x)
    scores.append(score)
return mean(scores)

Common pitfalls

• Varying numbers of contrastive sentences across dialects prevent direct cross-dialect score comparison without subsampling.
• For languages with empty dialect intersections (e.g., Italian), scores are derived from random 100-sentence samples averaged over 100 runs, introducing sampling variance.
• Using the standard-variety output as a pseudo-reference assumes the model translates the standard input perfectly, which may not hold for weak models.

Evidence (verbatim from paper)

The core of the idea is that we can treat $\tilde{\mathbf{y}}$, the output of the MT system on the "standard" input, as a pseudo-reference for the translation. Intuitively, a robust system should produce the same output for inputs with similar meanings regardless of the small dialectal variations. Hence, we can calculate any MT metric such as BLEU or COMET by comparing $\mathbf{y}$ to $\tilde{\mathbf{y}}$.

Citation

@misc{alam2023codet,
  title={CODET: A Benchmark for Contrastive Dialectal Evaluation of Machine Translation},
  author={Alam et al. (2023)},
  year={2023},
  note={arXiv:2305.17267}
}

• arXiv: 2305.17267