Qe4pe Eval

qe4pe-eval

QE4PE: Word-level Quality Estimation for Human Post-Editing — Sarti et al. (2025) (arXiv:2503.03044, 2025)

What this evaluates

Probes the practical usability and impact of word-level quality estimation highlights on professional translators' post-editing efficiency, accuracy, and workflow. It measures how different highlight modalities (oracle, supervised, unsupervised, none) affect editing effort, productivity, and final translation quality in real-world domain-specific settings.

Datasets

• QE4PE — total 413; splits: pre (38), main (325), post (50); repo https://github.com/gsarti/qe4pe

Metrics

• ESA score (primary) — range: [0, 100]
  • 0-100 segment-level quality score derived from the Direct Assessment (DA) protocol, where annotators rate the quality of post-edits relative to MT outputs and oracle references.
• Average Precision (AP) — range: [0, 1]
  • Threshold-agnostic metric calculating the area under the precision-recall curve for predicted error spans against oracle consensus spans.

Input / output format

Input: English source text + machine-translated output (optionally annotated with word-level error highlights indicating minor/major severity).

Output: Post-edited target text + behavioral logs (keystrokes, timestamps, highlight interactions) + MQM error annotations (severity, category, corrections) + ESA quality score.

Scoring recipe

def compute_ap(predictions, gold):
    tp, fp, fn = 0, 0, len(gold)
    for pred in sorted(predictions, key=lambda x: x[2], reverse=True):
        if any(span_iou(pred, g) > 0.5 for g in gold):
            tp += 1
            fn -= 1
        else:
            fp += 1
    precisions = [tp / (tp + fp) for tp, fp in zip(range(1, tp+1), range(fp, fp+tp+1))]
    return sum(precisions) / len(precisions) if precisions else 0.0

Common pitfalls

• The bare-bones GroTE interface lacks standard industry features (TMs, glossaries, spellcheck), which may not reflect real-world post-editing efficiency.
• Mixing Social and Biomedical domains in the same task reduces ecological validity, though it tests cross-domain robustness.
• Oracle highlights are generated from a 3-annotator consensus, which may overestimate typical human QE accuracy.

Evidence (verbatim from paper)

We use average precision (AP) as a threshold-agnostic performance metric for the tested continuous methods. Oracle highlights obtained from the consensus of three annotator in the first stage of the study are used as reference for QE4PE, while a single set of post-edits is available for DivEMT.

Citation

@misc{sarti2025qe4pe,
  title={QE4PE: Word-level Quality Estimation for Human Post-Editing},
  author={Sarti et al. (2025)},
  year={2025},
  note={arXiv:2503.03044}
}

• arXiv: 2503.03044