Nested Ner Historical Docs Eval

nested-ner-historical-docs-eval

A Benchmark of Nested Named Entity Recognition Approaches in Historical Structured Documents — Tual et al. (2023) (arXiv:2302.10204, 2023)

What this evaluates

This benchmark evaluates nested named entity recognition models on 19th-century Paris trade directories, testing their ability to extract hierarchical entities (e.g., addresses containing street names and numbers) and their robustness to OCR noise. It specifically probes how different sequence tagging formats (IO vs IOB2) and pre-training strategies affect span detection, hierarchical containment, and flat entity recognition.

Datasets

• Paris Trade Directories NER — total 8445; splits: (unstated)

Metrics

• F1-score (primary) — range: [0, 1]
  • Computed using the seqeval library by grouping consecutive tokens of the same class into predicted entities and aligning them with ground-truth spans. Any mismatch in boundary or class at any hierarchy level counts as an error. F1 is the harmonic mean of precision and recall across all evaluated entity levels (All, L1, L2, L1+L2, P-L1+P-L2, Flat).

Input / output format

Input: Tokenized text sequences from historical directory entries, formatted with either IO or IOB2 tagging schemes.

Output: Sequence of token-level labels (IO or IOB2) corresponding to entity types (PER, ACT, SPAT, LOC, CARDINAL, etc.) and hierarchy levels (1 or 2).

Scoring recipe

def compute_f1(pred_tags, gold_tags):
    pred_entities = group_consecutive_same_class(pred_tags)
    gold_entities = group_consecutive_same_class(gold_tags)
    tp = count_exact_matches(pred_entities, gold_entities)
    fp = len(pred_entities) - tp
    fn = len(gold_entities) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Using the IO tagging format without realizing it cannot distinguish adjacent entities of the same type, which artificially lowers performance compared to IOB2.
• Evaluating nested entities as flat labels, which ignores the Part-Of hierarchy and fails to capture the L1+L2 containment metric.
• Assuming the noisy OCR dataset retains all original entity spans; annotations are projected via alignment, causing some entries to be dropped (8445 vs original 8765).

Evidence (verbatim from paper)

We use the seqeval [18] library to evaluate the performance of each approach. It is well suited to the evaluation of natural language processing tasks, including sequence labelling. The tool supports the IO and IOB2 tag formats. The metrics used for the evaluation are precision, recall, and the F1-score. To compute these values, the tool first gathers tokens of the same class that follow each other. These groups of tokens constitute the predicted entities that are aligned with the ground-truth. Any difference in the boundary of the entity or its class, at any level, will be considered as an error.

Citation

@misc{tual2023nested,
  title={A Benchmark of Nested Named Entity Recognition Approaches in Historical Structured Documents},
  author={Tual et al. (2023)},
  year={2023},
  note={arXiv:2302.10204}
}

• arXiv: 2302.10204