# Nested Ner Eval

> Evaluates a model's ability to identify and classify named entities that can overlap or be contained within other entities (nested NER) across multiple domains. It probes the model's span-level understanding and label assignment capabilities in complex textual contexts. Use when the user wants to benchmark on ACE2004, ACE2005, GENIA, KBP2017, or asks about evaluating this task. Reports F1.

- Skill: `qhjqhj00/nested-ner-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/nested-ner-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/nested-ner-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/nested-ner-eval

---


# nested-ner-eval

> Fusing Heterogeneous Factors with Triaffine Mechanism for Nested Named Entity Recognition — Zheng Yuan et al. (2021) (arXiv:2110.07480, 2021)

## What this evaluates

Evaluates a model's ability to identify and classify named entities that can overlap or be contained within other entities (nested NER) across multiple domains. It probes the model's span-level understanding and label assignment capabilities in complex textual contexts.

## Datasets

- **ACE2004** — total ?; splits: train (6200), dev (745), test (812)
- **ACE2005** — total ?; splits: train (7194), dev (969), test (1047)
- **GENIA** — total ?; splits: train (16692), test (1854)
- **KBP2017** — total ?; splits: train (10546), dev (545), test (4267)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Computed at the span level: Precision = |correctly predicted spans| / |predicted spans|, Recall = |correctly predicted spans| / |gold spans|, F1 = 2 * Precision * Recall / (Precision + Recall). Only exact matches of span boundaries and entity labels count as correct.

## Input / output format

**Input**: Tokenized sentence with contextual embeddings (BERT/BioBERT), supplemented by fastText word embeddings, POS embeddings, and character-level BiLSTM embeddings. Spans are implicitly formed by token indices.

**Output**: A set of predicted spans, each defined by a start token index, an end token index, and a predicted entity label.

## Scoring recipe

```python
def compute_span_f1(preds, golds):
    correct = sum(1 for p in preds if p in golds)
    precision = correct / len(preds) if preds else 0
    recall = correct / len(golds) if golds else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1
```

## Common pitfalls

- Using token-level or sequence-labeling metrics (e.g., BIO accuracy) instead of exact span-level matching for nested entities.
- Ignoring the specific dataset splits mandated by prior works (Lu & Roth 2015 for ACE, Lin et al. 2019 for GENIA/KBP), which differ from standard LDC splits.
- Failing to handle overlapping/nested spans correctly during evaluation, leading to inflated or deflated scores if not matched exactly by (start, end, label) tuples.

## Evidence (verbatim from paper)

> Following previous work, we measure the results using span-level precision, recall, and $F_{1}$ scores.

## Citation

```bibtex
@misc{yuan2021triaffine,
  title={Fusing Heterogeneous Factors with Triaffine Mechanism for Nested Named Entity Recognition},
  author={Zheng Yuan et al. (2021)},
  year={2021},
  note={arXiv:2110.07480}
}
```

- arXiv: 2110.07480