Punctuation Restoration Iwslt Eval

punctuation-restoration-iwslt-eval

Adversarial Transfer Learning for Punctuation Restoration — Jiangyan Yi et al. (2020) (arXiv:2004.00248, 2020)

What this evaluates

Evaluates a model's ability to restore punctuation marks (commas, periods, question marks) in English text. It specifically probes robustness on both manually transcribed transcripts and ASR-generated transcripts, ignoring non-punctuation tokens during evaluation.

Datasets

• IWSLT2011 — total ?; splits: train (-1), dev (-1), test_Ref (-1), test_ASR (-1)

Metrics

• F1-score (primary) — range: percent
  • Standard F1 score calculated as 2 * (Precision * Recall) / (Precision + Recall). Precision and Recall are computed exclusively on punctuation tokens (COMMA, PERIOD, QUESTION), completely ignoring non-punctuation tokens (O).
• Precision — range: percent
  • Ratio of correctly predicted punctuation tokens to all predicted punctuation tokens. Computed only on COMMA, PERIOD, and QUESTION labels.
• Recall — range: percent
  • Ratio of correctly predicted punctuation tokens to all actual punctuation tokens. Computed only on COMMA, PERIOD, and QUESTION labels.

Input / output format

Input: A sequence of English words/tokens from TED Talk transcripts.

Output: A sequence of token-level labels: O, COMMA, PERIOD, or QUESTION.

Scoring recipe

def compute_metrics(predictions, gold):
    # Filter out non-punctuation tokens (O) as per protocol
    pred_punct = [p for p, g in zip(predictions, gold) if g != 'O']
    gold_punct = [g for g in gold if g != 'O']
    
    tp = sum(1 for p, g in zip(pred_punct, gold_punct) if p == g)
    fp = sum(1 for p, g in zip(pred_punct, gold_punct) if p != g)
    fn = sum(1 for p, g in zip(pred_punct, gold_punct) if p != g)
    
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1

Common pitfalls

• Including non-punctuation tokens (label 'O') in the precision/recall/F1 calculation, which artificially inflates scores.
• Failing to distinguish between the two test sets (Ref. for manual transcripts vs. ASR for speech recognition outputs), leading to misreported results.
• Misinterpreting the label mapping: exclamation marks and semicolons are grouped under PERIOD, not treated as separate classes.

Evidence (verbatim from paper)

All models are evaluated in terms of precision (P), recall (R), F1-score (F1) in our experiments. We focus on the performance of the punctuation marks. So the correctly predicted non-punctuation marks O are ignored. We only evaluate the performance of COMMA, PERIOD and QUESTION on two test sets: Ref. and ASR, respectively.

Citation

@misc{yi2020adversarial,
  title={Adversarial Transfer Learning for Punctuation Restoration},
  author={Jiangyan Yi et al. (2020)},
  year={2020},
  note={arXiv:2004.00248}
}

• arXiv: 2004.00248