Quality Classification Eval

quality-classification-eval

Practical Perspectives on Quality Estimation for Machine Translation — Zhou et al. (2020) (arXiv:2005.03519, 2020)

What this evaluates

Evaluates a model's ability to classify machine translation outputs as 'good' (zero HTER) or 'bad' (non-zero HTER) for practical post-editing filtering. It probes whether binary classification outperforms thresholded regression for identifying adequate translations in real-world deployment scenarios.

Datasets

• WMT17 QE/QC — total ?; splits: train (48000), dev (2000), test (4000)

Metrics

• R@P_t (primary) — range: [0, 1]
  • Recall at precision above threshold t. Computes the recall achieved on the positive class when the model's precision on predicted positives is at least t. The paper evaluates at t=0.8 and t=0.9.

Input / output format

Input: Source sentence and target sentence pair.

Output: Binary label: 'good' (positive) or 'bad' (negative).

Scoring recipe

def compute_R_at_Pt(preds, gold, t):
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    if precision >= t:
        recall = tp / sum(gold) if sum(gold) > 0 else 0.0
        return recall
    return 0.0

Common pitfalls

• Class imbalance is severe, as only sentences with exactly 0.0 HTER are labeled positive (9–44% positive rate depending on language pair and split).
• Thresholding a regression model's TER output to create a binary classifier yields poor precision/recall trade-offs compared to training a dedicated binary classifier from scratch.
• Performance varies significantly between language directions (En-De vs De-En) due to domain mismatch with the parallel training data (IT vs. Pharmaceutical).

Evidence (verbatim from paper)

QE datasets list source/target sentence pairs with HTER scores as labels; for QC we label samples with 0.0 HTER as 'good' (positive) while the rest get 'bad' (negative) labels. We have tuned hyperparameters for QC models according to the $R@P_t$ on the development dataset by grid-search, and the final parameters we finally picked are shown in Table 2.

Citation

@misc{zhou2020practical,
  title={Practical Perspectives on Quality Estimation for Machine Translation},
  author={Zhou et al. (2020)},
  year={2020},
  note={arXiv:2005.03519}
}

• arXiv: 2005.03519