spaCy-ner-eval

spaCy-ner-eval

Multi hash embeddings in spaCy — Miranda et al. (2022) (arXiv:2212.09255, 2022)

What this evaluates

Evaluates named entity recognition performance across diverse domains and languages, specifically probing a model's ability to handle out-of-vocabulary words and morphological variation through hash-based embeddings versus traditional lookup embeddings.

Datasets

• CoNLL 2002 — total ?; splits: train (-1), val (-1), test (-1)
• WNUT 2017 — total ?; splits: train (-1), test (-1)
• AnEM — total ?; splits: train (-1), dev (-1), test (-1)
• Dutch Archaeology — total ?; splits: train (-1), dev (-1), test (-1)
• OntoNotes 5.0 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Standard token-level F1 score computed over exact entity span matches: F1 = 2 * (precision * recall) / (precision + recall), where precision and recall are calculated over predicted vs. gold entity spans.

Input / output format

Input: Tokenized text with sentence/document boundaries and gold entity annotations.

Output: BILUO sequence tags (Begin, In, Last, Unit, Out) for each token, mapping to entity types.

Scoring recipe

def compute_f1(gold_spans, pred_spans):
    tp = len(gold_spans & pred_spans)
    precision = tp / len(pred_spans) if pred_spans else 0.0
    recall = tp / len(gold_spans) if gold_spans else 0.0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0

Common pitfalls

• Unseen entity evaluation strictly ignores spans verbatim present in training, which differs from standard exact-match or substring matching.
• Custom random splits for AnEM and Dutch Archaeology deviate from canonical benchmarks, limiting direct comparability.
• Document-level segmentation for Dutch CoNLL 2002 vs sentence-level for others affects context availability.

Evidence (verbatim from paper)

Unseen entities are evaluated by ignoring all known entity spans during evaluation. Evaluated on the test set.

Citation

@misc{miranda2022multihash,
  title={Multi hash embeddings in spaCy},
  author={Miranda et al. (2022)},
  year={2022},
  note={arXiv:2212.09255}
}

• arXiv: 2212.09255