Ist Unbabel 2022 Qe Eval

ist-unbabel-2022-qe-eval

CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task — Rei et al. (2022) (arXiv:2209.06243, 2022)

What this evaluates

Evaluates machine translation quality estimation (QE) by predicting human quality scores at the sentence level and identifying error locations at the word level. It also assesses the model's ability to generate faithful explanations for predicted errors.

Datasets

• IST-Unbabel 2022 QE Shared Task — total ?; splits: internal_dev (-1), internal_test (-1); repo https://github.com/Unbabel/COMET

Metrics

• Spearman's rank correlation (primary) — range: [-1, 1]
  • Computes Pearson correlation between the ranks of predicted quality scores and gold human assessment scores.
• Matthew's correlation coefficient (MCC) (primary) — range: [-1, 1]
  • Standard MCC formula for binary classification distinguishing error words from non-error words at the word level.
• Recall@K (R@K) (primary) — range: [0, 1]
  • Proportion of actual error words correctly identified among the top-K predicted error words, evaluated only on translations containing errors.

Input / output format

Input: Source sentence, target sentence, and reference translation. For word-level and explainability tasks, gold word-level error tags or continuous quality scores are provided during training.

Output: Continuous quality score per sentence; binary error tag or continuous score per word; continuous explanation score per word.

Scoring recipe

def spearman_corr(pred, gold):
    pred_rank = rankdata(pred)
    gold_rank = rankdata(gold)
    return pearsonr(pred_rank, gold_rank)[0]

def mcc_score(pred_tags, gold_tags):
    return matthews_corrcoef(gold_tags, pred_tags)

def recall_at_k(pred_scores, gold_tags, k=5):
    top_k = np.argsort(pred_scores)[-k:]
    true_errors = np.where(gold_tags == 1)[0]
    return len(set(top_k) & set(true_errors)) / len(true_errors)

Common pitfalls

• Internal dev/test splits are created by halving the provided development set, which differs from the official Codalab test set used for final ranking.
• MQM and DA data exhibit high variance and different characteristics; combining them requires careful concatenation or ensembling to avoid performance drops.
• Directionality of language pairs significantly impacts performance, with out-of-English translations often showing different trends than in-English ones.

Evidence (verbatim from paper)

Sentence-level submissions were evaluated using the Spearman's rank correlation. Pearson's correlation, MAE, and RMSE were also used as secondary metrics, but here we report only Spearman correlation since it was the primary metric used to rank systems. Word-level submission were evaluated using MCC, F1-OK, and F1-BAD, but we report only MCC as it was considered the main metric.

Citation

@misc{rei2022cometkiwi,
  title={CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task},
  author={Rei et al. (2022)},
  year={2022},
  note={arXiv:2209.06243}
}

• arXiv: 2209.06243