Long Ner Eval

long-ner-eval

Regularization for Long Named Entity Recognition — Minbyul Jeong, Jaewoo Kang (2021) (arXiv:2104.07249, 2021)

What this evaluates

Evaluates named entity recognition capabilities, specifically probing a model's ability to handle class imbalance, out-of-vocabulary terms, and long or complex entity names across biomedical and general domain texts.

Datasets

• NCBI-disease — total ?; splits: train (-1), val (-1), test (-1)
• BC5CDR-disease — total ?; splits: train (-1), val (-1), test (-1)
• BC5CDR-chemical — total ?; splits: train (-1), val (-1), test (-1)
• BC4CHEMD — total ?; splits: train (-1), val (-1), test (-1)
• BC2GM — total ?; splits: train (-1), val (-1), test (-1)
• JNLPBA — total ?; splits: train (-1), val (-1), test (-1)
• LINNAEUS — total ?; splits: train (-1), val (-1), test (-1)
• Species-800 — total ?; splits: train (-1), val (-1), test (-1)
• CoNLL-2003 — total ?; splits: train (-1), val (-1), test (-1)
• WNUT-2017 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall calculated over all predicted entity mentions. Precision is the fraction of predicted entities that are correct, and recall is the fraction of gold entities that are correctly predicted.
• Recall — range: [0, 1]
  • Fraction of gold standard entity mentions correctly identified by the model. The paper partitions recall into Memorization (Mem), Synonym (Syn), Concept (Con), and Unseen categories to evaluate generalization.

Input / output format

Input: Tokenized text sentences with corresponding token-level entity labels.

Output: Token-level BIO/IOB entity labels predicting the entity type and boundary for each token in the input sequence.

Scoring recipe

def calculate_ner_f1(preds, gold):
    pred_set = set(extract_mentions(preds))
    gold_set = set(extract_mentions(gold))
    tp = len(pred_set & gold_set)
    fp = len(pred_set - gold_set)
    fn = len(gold_set - pred_set)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0

def extract_mentions(labels):
    mentions = set()
    curr_type, start = None, None
    for i, lbl in enumerate(labels):
        if lbl.startswith('B-'):
            if curr_type: mentions.add((curr_type, start, i-1))
            curr_type, start = lbl[2:], i
        elif lbl.startswith('I-') and curr_type:
            pass
        else:
            if curr_type: mentions.add((curr_type, start, i-1))
            curr_type, start = None, None
    if curr_type: mentions.add((curr_type, start, len(labels)-1))
    return mentions

Common pitfalls

• The paper partitions datasets into Memorization (Mem), Synonym (Syn), Concept (Con), and Unseen categories; reporting only aggregate F1 masks performance differences across these generalization splits.
• Entity length heavily influences results; models often perform well on short entities but drop significantly on long entities (≥8 tokens), requiring length-stratified evaluation.
• Out-of-vocabulary (OOV) handling relies on subword tokenization rather than word-level frequency, so evaluating at the word level without considering subword debiasing will misrepresent OOV performance.

Evidence (verbatim from paper)

Using three components, namely Subword, Class, and Temp, showed significant improvements in recall on Syn and Con, as well as overall improvements to in-domain performance (F1).

Citation

@misc{jeong2021regularization,
  title={Regularization for Long Named Entity Recognition},
  author={Minbyul Jeong, Jaewoo Kang (2021)},
  year={2021},
  note={arXiv:2104.07249}
}

• arXiv: 2104.07249