Bert Noise Robustness Eval

bert-noise-robustness-eval

Noisy Text Data: Achilles' Heel of BERT — Ankit Kumar et al. (2020) (arXiv:2003.12932, 2020)

What this evaluates

Evaluates BERT's robustness to synthetic character-level noise across sentiment classification and textual similarity tasks. It probes how spelling mistakes and typos disrupt subword tokenization and degrade contextual embeddings under varying noise intensities.

Datasets

• IMDB — total 50000; splits: train (25000), test (25000)
• SST-2 — total 68221; splits: train (67349), test (872)
• STS-B — total 7249; splits: train (5749), test (1500)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
• Pearson-Spearman correlation — range: [-1, 1]
  • Measures the monotonic relationship between predicted similarity scores and ground truth scores (0–5).

Input / output format

Input: Text inputs (single sentences or sentence pairs) with synthetic character-level noise injected at specified percentages (2.5%–22.5%).

Output: Binary sentiment label (positive/negative) or continuous similarity score (0–5).

Scoring recipe

def compute_metrics(preds, gold, task):
    if task == 'sentiment':
        tp = sum(1 for p, g in zip(preds, gold) if p == g == 1)
        fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
        fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    elif task == 'similarity':
        return pearsonr(preds, gold)[0]

Common pitfalls

• Noise is injected at the character level (random QWERTY neighbor replacement), not word-level, which heavily impacts BERT's subword tokenizer.
• The noise percentage refers to the proportion of characters replaced per instance, not words or sentences.
• Performance drops are reported as accuracy/error in figures but F1/correlation in text; readers must distinguish between the two reporting formats.

Evidence (verbatim from paper)

To measure the performance of the model for sentiment analysis task we use F1 score. ... Here, we use Pearson-Spearman correlation to measure model's performance.

Citation

@misc{kumar2020noisytext,
  title={Noisy Text Data: Achilles' Heel of BERT},
  author={Ankit Kumar et al. (2020)},
  year={2020},
  note={arXiv:2003.12932}
}

• arXiv: 2003.12932