Ner Framework Eval

ner-framework-eval

Recent Advances in Named Entity Recognition: A Comprehensive Survey and Comparative Study — Keraghel et al. (2024) (arXiv:2401.10825, 2024)

What this evaluates

This evaluation protocol benchmarks Named Entity Recognition (NER) systems across diverse domains and entity type distributions. It measures how well different architectures (transformers, CRFs, LLMs) identify and classify named entity spans under exact-match conditions.

Datasets

• CoNLL-2003 — total ?; splits: test (-1)
• OntoNotes — total ?; splits: test (-1)
• WNUT2017 — total ?; splits: test (-1)
• FIN — total ?; splits: test (-1)
• BioNLP2004 — total ?; splits: test (-1)
• NCBI Disease — total ?; splits: test (-1)
• BC5CDR — total ?; splits: test (-1)
• MITRestaurant — total ?; splits: test (-1)
• Few-NERD — total ?; splits: test (-1)
• MultiCoNER — total ?; splits: test (-1)

Metrics

• Macro-averaged F1-score (primary) — range: [0, 1]
  • Computed as the harmonic mean of precision and recall across all entity types, then averaged across types. Evaluated using exact match of entity spans and labels.

Input / output format

Input: Raw text converted to CoNLL-U format (BIO scheme), then adapted to framework-specific formats. For GPT-4, text is wrapped in custom prompts specifying target entity categories with few-shot examples.

Output: BIO-tagged entity spans for traditional models. For GPT-4, JSON-formatted entity extraction results.

Scoring recipe

def compute_ner_f1(pred_entities, gold_entities):
    correct = sum(1 for p in pred_entities if p in gold_entities)
    precision = correct / len(pred_entities) if pred_entities else 0.0
    recall = correct / len(gold_entities) if gold_entities else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1

Common pitfalls

• Apache OpenNLP only supports three entity types (persons, organizations, locations), leading to missing results on datasets lacking these categories.
• GPT-4 uses a unified, less-directed prompting strategy rather than type-specific prompts, which intentionally makes disambiguation harder and may lower scores compared to prior LLM-NER studies.
• Missing OpenNLP scores are imputed as zeros for the Friedman/Nemenyi statistical tests, which can artificially depress aggregate rankings.

Evidence (verbatim from paper)

To assess the quality of the results we use a strategy of exact evaluation. Our chosen metric is F1-score, since this reflects the two other metrics discussed in section 7.3 (namely, precision and recall).

Citation

@misc{keraghel2024recent,
  title={Recent Advances in Named Entity Recognition: A Comprehensive Survey and Comparative Study},
  author={Keraghel et al. (2024)},
  year={2024},
  note={arXiv:2401.10825}
}

• arXiv: 2401.10825