Span Mt Metaeval

span-mt-metaeval

Span-Level Machine Translation Meta-Evaluation — Perrella et al. (2026) (arXiv:2603.19921, 2026)

What this evaluates

Evaluates the reliability and fairness of span-level error detection metrics for machine translation auto-evaluators. It probes whether standard micro-averaged precision/recall/F1 scores produce consistent rankings compared to a proposed partial overlap matching strategy.

Datasets

• MQM 2022-2024 — total ?; splits: test (-1)

Metrics

• micro-averaged precision/recall/F1 (primary) — range: [0, 1]
  • Computed as the harmonic mean of micro-averaged precision and recall across all predicted and gold span-level errors. Precision is the fraction of predicted spans that match gold spans, and recall is the fraction of gold spans matched by predictions.
• mpp — range: [0, 1]
  • Match with partial overlap and partial credit, using micro-averaging. Assigns fractional credit based on the degree of overlap between predicted and gold spans rather than requiring exact matches.

Input / output format

Input: Source sentence, machine-translated hypothesis, reference sentence, and gold-standard span-level error annotations.

Output: Predicted span-level error annotations (start/end offsets and error types) for the hypothesis.

Scoring recipe

def compute_micro_f1(pred_spans, gold_spans):
    tp = sum(1 for p in pred_spans if p in gold_spans)
    fp = len(pred_spans) - tp
    fn = len(gold_spans) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1

Common pitfalls

• Exact span matching is often too strict for MT error detection, leading to artificially low scores.
• Micro-averaging across diverse error types or language pairs can mask model weaknesses in specific categories.
• Arbitrary methodological choices in span alignment cause inconsistent auto-evaluator rankings.

Evidence (verbatim from paper)

common metrics like micro-averaged precision/recall/F1 produce inconsistent rankings due to arbitrary methodological choices. We demonstrate that some meta-evaluation strategies are unsuitable for MT error detection by measuring the performance of sentinel auto-evaluators alongside normal auto-evaluators.

Citation

@misc{perrella2026spanmtmetaeval,
  title={Span-Level Machine Translation Meta-Evaluation},
  author={Perrella et al. (2026)},
  year={2026},
  note={arXiv:2603.19921}
}

• arXiv: 2603.19921