Nested Ner Eval

nested-ner-eval

Fusing Heterogeneous Factors with Triaffine Mechanism for Nested Named Entity Recognition — Zheng Yuan et al. (2021) (arXiv:2110.07480, 2021)

What this evaluates

Evaluates a model's ability to identify and classify named entities that can overlap or be contained within other entities (nested NER) across multiple domains. It probes the model's span-level understanding and label assignment capabilities in complex textual contexts.

Datasets

• ACE2004 — total ?; splits: train (6200), dev (745), test (812)
• ACE2005 — total ?; splits: train (7194), dev (969), test (1047)
• GENIA — total ?; splits: train (16692), test (1854)
• KBP2017 — total ?; splits: train (10546), dev (545), test (4267)

Metrics

• F1 (primary) — range: [0, 1]
  • Computed at the span level: Precision = |correctly predicted spans| / |predicted spans|, Recall = |correctly predicted spans| / |gold spans|, F1 = 2 * Precision * Recall / (Precision + Recall). Only exact matches of span boundaries and entity labels count as correct.

Input / output format

Input: Tokenized sentence with contextual embeddings (BERT/BioBERT), supplemented by fastText word embeddings, POS embeddings, and character-level BiLSTM embeddings. Spans are implicitly formed by token indices.

Output: A set of predicted spans, each defined by a start token index, an end token index, and a predicted entity label.

Scoring recipe

def compute_span_f1(preds, golds):
    correct = sum(1 for p in preds if p in golds)
    precision = correct / len(preds) if preds else 0
    recall = correct / len(golds) if golds else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1

Common pitfalls

• Using token-level or sequence-labeling metrics (e.g., BIO accuracy) instead of exact span-level matching for nested entities.
• Ignoring the specific dataset splits mandated by prior works (Lu & Roth 2015 for ACE, Lin et al. 2019 for GENIA/KBP), which differ from standard LDC splits.
• Failing to handle overlapping/nested spans correctly during evaluation, leading to inflated or deflated scores if not matched exactly by (start, end, label) tuples.

Evidence (verbatim from paper)

Following previous work, we measure the results using span-level precision, recall, and $F_{1}$ scores.

Citation

@misc{yuan2021triaffine,
  title={Fusing Heterogeneous Factors with Triaffine Mechanism for Nested Named Entity Recognition},
  author={Zheng Yuan et al. (2021)},
  year={2021},
  note={arXiv:2110.07480}
}

• arXiv: 2110.07480