Cti Plausibility Eval

cti-plausibility-eval

Quantifying the Plausibility of Context Reliance in Neural Machine Translation — Sarti et al. (2023) (arXiv:2310.01188, 2023)

What this evaluates

Evaluates whether neural machine translation models correctly rely on contextual cues when generating target tokens. It compares model-extracted cue-target pairs against human-annotated discourse-level expectations to measure the plausibility of context reliance.

Datasets

• SCAT+ — total ?; splits: test (-1)

Metrics

• Macro F1 (primary) — range: [0, 1]
  • Standard macro-averaged F1 score computed over binary predictions of context-sensitive tokens versus gold SCAT+ labels.
• AUPRC — range: [0, 1]
  • Area Under the Precision-Recall Curve, measuring the trade-off between precision and recall across different thresholds for identifying context-dependent tokens.

Input / output format

Input: Source context (Cx), target context (Cy), source sentence (x), and target sentence (y) for machine translation instances.

Output: Binary predictions or scores indicating whether specific target tokens are context-sensitive, aligned with gold SCAT+ annotations.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold_labels) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold_labels) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return {"macro_f1": f1, "auprc": compute_auprc(gold_labels, predictions)}

Common pitfalls

• SCAT+ annotations only cover gender disambiguation for anaphoric pronouns, missing other context dependencies like formality cohesion.
• The evaluation is explicitly noted as a lower bound because it is restricted to only two phenomena (anaphora resolution and lexical choice) available in the dataset.

Evidence (verbatim from paper)

CTI Results Figure 5 and Figure 6 present the CTI plausibility of all tested models for the Macro F1 and AUPRC metrics, similarly to Figure 3 in the main analysis. This suggests our evaluation of CTI metrics plausibility can be considered a lower bound, as it is restricted to the two phenomena available in the datasets we used (anaphora resolution and lexical choice).

Citation

@misc{sarti2023quantifying,
  title={Quantifying the Plausibility of Context Reliance in Neural Machine Translation},
  author={Sarti et al. (2023)},
  year={2023},
  note={arXiv:2310.01188}
}

• arXiv: 2310.01188