Weakly Supervised Ner Eval

weakly-supervised-ner-eval

BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognition — Yinghao Li et al. (arXiv:2105.12848, 2021)

What this evaluates

This evaluation probes a model's ability to perform named entity recognition under weak supervision, where training labels are noisy and derived from multiple distant supervision sources. It measures how well the model can denoise these labels and generalize entity patterns across general, biomedical, and review domains.

Datasets

• CoNLL 2003 — total 22137; splits: train (14041), dev (3250), test (3453)
• LaptopReview — total 3845; splits: train (2436), dev (609), test (800)
• NCBI-Disease — total 793; splits: train (593), dev (100), test (100)
• BC5CDR — total 1500; splits: train (500), dev (500), test (500)

Metrics

• entity-level F1 (primary) — range: percent
  • F1 = 2 * (Precision * Recall) / (Precision + Recall), computed at the exact entity span level. Precision is the fraction of predicted entities that exactly match gold spans, and Recall is the fraction of gold entities correctly predicted.

Input / output format

Input: Raw sentences (word tokens) accompanied by multiple noisy weak supervision label sources per token.

Output: Sequence of entity labels per token (e.g., BIO tags) or a list of predicted entity spans with their types.

Scoring recipe

def entity_f1(preds, golds):
    pred_spans = extract_spans(preds)
    gold_spans = extract_spans(golds)
    tp = len(set(pred_spans) & set(gold_spans))
    fp = len(set(pred_spans) - set(gold_spans))
    fn = len(set(gold_spans) - set(pred_spans))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return f1 * 100

Common pitfalls

• Evaluating at the token level instead of the entity span level, which inflates scores for NER tasks.
• Reporting single-run results instead of averaging over 5 different random seeds as specified in the protocol.
• Ignoring the multi-source weak supervision setup and treating the task as standard fully supervised NER.

Evidence (verbatim from paper)

We evaluate the performance of NER models using entity-level precision, recall, and F1 scores. All scores are presented as percentages. The results come from the average of 5 trials with different random seeds.

Citation

@misc{li2021bertifying,
  title={BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognition},
  author={Yinghao Li et al.},
  year={2021},
  note={arXiv:2105.12848}
}

• arXiv: 2105.12848