Ner Eval

ner-eval

Simple Questions Generate Named Entity Recognition Datasets — Kim et al. (2021) (arXiv:2112.08808, 2021)

What this evaluates

Evaluates named entity recognition (NER) models under data-scarce conditions, specifically low-resource settings with no human-annotated training labels and few-shot settings with minimal labeled examples. It probes the model's ability to identify and classify entity types in text using automatically generated pseudo-dictionaries and weak supervision.

Datasets

• CoNLL-2003 — total ?; splits: test (-1)
• Wikigold — total ?; splits: test (-1)
• WNUT-16 — total ?; splits: test (-1)
• NCBI-disease — total ?; splits: test (-1)
• BC5CDR — total ?; splits: test (-1)
• CHEMDNER — total ?; splits: test (-1)

Metrics

• F1score (primary) — range: [0, 1]
  • Entity-level F1 score computed from precision and recall over all correctly identified entity spans. Macro-averaged across datasets in low-resource experiments.

Input / output format

Input: Raw text sequences (sentences, abstracts, or tweets) for token-level entity prediction. In few-shot settings, a small set of labeled training examples (10-20 sentences) is also provided for fine-tuning.

Output: Token-level entity type labels indicating the class and boundaries of named entities in the input text.

Scoring recipe

def compute_ner_f1(pred_spans, gold_spans):
    pred_set = set(pred_spans)
    gold_set = set(gold_spans)
    tp = len(pred_set & gold_set)
    fp = len(pred_set - gold_set)
    fn = len(gold_set - pred_set)
    p = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    r = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * p * r / (p + r) if (p + r) > 0 else 0.0

Common pitfalls

• The paper excludes 'miscellaneous' entity types in low-resource experiments but includes them in few-shot experiments for fair baseline comparison, which can cause score inconsistencies if not handled carefully.
• Hyperparameters and model checkpoints are selected using validation sets due to the absence of human-annotated training labels, deviating from standard train/val/test splits.
• Entity subtype definitions vary significantly across datasets (e.g., 'organization' in CoNLL-2003 vs Wikigold), requiring dataset-specific query formulation rather than a one-size-fits-all dictionary.

Evidence (verbatim from paper)

We used entity-level precision (P), recall (R), and F1score (F1) as the evaluation metrics.

Citation

@misc{kim2021simple,
  title={Simple Questions Generate Named Entity Recognition Datasets},
  author={Kim et al. (2021)},
  year={2021},
  note={arXiv:2112.08808}
}

• arXiv: 2112.08808