Noisyner Eval

noisyner-eval

Analysing the Noise Model Error for Realistic Noisy Label Data — Hedderich et al. (2021) (arXiv:2101.09763, 2021)

What this evaluates

Evaluates the robustness of noise models and base models under realistic noisy label conditions, measuring how estimation accuracy and base model performance vary with different noise distributions and amounts of clean data.

Datasets

• NoisyNER — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/uds-lsv/noise-estimation

Metrics

• micro-average F1 score (primary) — range: [0, 1]
  • Micro-averaged F1 score computed over all classes, excluding the non-entity label. Calculated as the harmonic mean of micro-precision and micro-recall across all instances.

Input / output format

Input: Token sequences with noisy labels for NER, alongside a small uniformly sampled clean subset $D_C$ for noise model estimation.

Output: Predicted entity labels for each token in the test set.

Scoring recipe

confusion = compute_confusion_matrix(predictions, gold_labels, exclude_non_entity=True)
precision = confusion.diag.sum() / confusion.sum(axis=0).sum()
recall = confusion.diag.sum() / confusion.sum(axis=1).sum()
f1 = 2 * precision * recall / (precision + recall + 1e-8)
return f1.mean()

Common pitfalls

• Excluding the non-entity label is critical; including it would severely skew the F1 score.
• Experiments must be repeated 50 times with different random seeds to report mean and standard deviation, as results vary significantly with sampling strategy.

Evidence (verbatim from paper)

We split the NoisyNER data into an 80/10/10 train/dev/test split. Following the standard approach for named entity recognition (Tjong Kim Sang & De Meulder, 2003), we evaluate with the micro-average F1 score excluding the non-entity label. Each experiment is repeated 50 times to report mean and standard deviation.

Citation

@misc{hedderich2021analysing,
  title={Analysing the Noise Model Error for Realistic Noisy Label Data},
  author={Hedderich et al. (2021)},
  year={2021},
  note={arXiv:2101.09763}
}

• arXiv: 2101.09763