Ssa Mte Eval

ssa-mte-eval

SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages? — Senyu Li et al. (2025) (arXiv:2506.04557, 2025)

What this evaluates

Evaluates machine translation quality estimation metrics on under-resourced African languages by comparing their predicted scores against human-annotated Direct Assessment (DA) judgments. It probes a model's ability to correlate with human perception of translation adequacy across diverse language pairs, including both reference-based and reference-free settings.

Datasets

• SSA-MTE — total 73000; splits: train (-1), dev (-1), test (-1)

Metrics

• Spearman correlation (primary) — range: [-1, 1]
  • Spearman rank correlation coefficient computed between the model's predicted quality scores and the human-annotated Direct Assessment (DA) scores across all test instances.

Input / output format

Input: Source sentence, reference translation (for MTE), and hypothesis translation. For QE, only source and hypothesis are provided.

Output: A continuous quality score normalized to the [0, 1] range.

Scoring recipe

def compute_spearman(predictions, gold_scores):
    from scipy.stats import spearmanr
    corr, _ = spearmanr(predictions, gold_scores)
    return corr

Common pitfalls

• Human DA scores must be pre-processed with z-score normalization per evaluator followed by min-max scaling to [0, 1] before training or evaluation to ensure comparability.
• LLM-based baselines require a 5-shot in-context prompt sampled from the training split; performance drops significantly without these demonstrations.

Evidence (verbatim from paper)

As shown in Table[3], SSA-COMET-MTL achieves the second highest average Spearman correlation with human judgments in the MTE setting, outperforming all prior AfriCOMET variants as well as the strong prompting-based baselines such as Gemini-2.5 Pro.

Citation

@misc{li2025ssacomet,
  title={SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?},
  author={Senyu Li et al. (2025)},
  year={2025},
  note={arXiv:2506.04557}
}

• arXiv: 2506.04557