Net Ner Probing Eval

net-ner-probing-eval

Probing Pre-trained Auto-regressive Language Models for Named Entity Typing and Recognition — Epure et al. (2021) (arXiv:2108.11857, 2021)

What this evaluates

Evaluates zero-shot and few-shot named entity typing (NET) and recognition (NER) capabilities of pre-trained auto-regressive language models without fine-tuning. Probes reliance on memorized lexical patterns versus contextual generalization and tests robustness to noisy text and case variations.

Datasets

• CoNLL-2003 — total ?; splits: test (-1)
• WNUT2017 — total ?; splits: test (-1)
• MIT Movie — total ?; splits: test (-1)
• DBpedia — total ?; splits: test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Macro-averaged F1 score computed over exact matches, with case-insensitive comparison and a normalized Levenshtein distance threshold of 0.2 for string similarity. Also includes a rule where generating a string with no common words when no entity is expected counts as correct.

Input / output format

Input: For NET: NE mention paired with type definitions/keywords. For NER: Few-shot prompt (16 examples: 9 positive, 7 negative) followed by a test sentence.

Output: For NET: Predicted entity type (selected via lowest perplexity). For NER: Generated string of up to 15 tokens representing the extracted entity.

Scoring recipe

def compute_f1(predictions, golds):
    tp, fp, fn = 0, 0, 0
    for pred, gold in zip(predictions, golds):
        pred_norm = pred.lower().replace(' ', '')
        gold_norm = gold.lower().replace(' ', '')
        if pred_norm == gold_norm:
            tp += 1
        elif normalized_levenshtein(pred, gold) / len(gold) < 0.2:
            tp += 1
        elif gold == 'none' and not any(word in pred.lower() for word in gold.split()):
            tp += 1
        else:
            fp += 1
            fn += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Case differences and extra spaces in generated entities are normalized rather than penalized, which can mask generation errors.
• Multiple entities of the same type in a single sentence are handled by a 'match any' rule, which can inflate scores if the model outputs only one.
• Memorization thresholds differ by dataset (word vs. transition exposure), requiring careful split construction that varies across datasets.

Evidence (verbatim from paper)

Table 2: Zero-shot NET F1-scores. ... In computing scores, we rely mostly on exact NE matching with some exceptions. The evaluation is insensitive to the letter case (e.g. 'none' and 'None' are considered equivalent). Also, we noticed that the model tends to add spaces for NEs written together such as in social media mentions. To cover these cases, we consider that the prediction is equal to the ground-truth, if their Levenshtein distance divided by the true NE length is lower than 0.2.

Citation

@misc{epure2021probing,
  title={Probing Pre-trained Auto-regressive Language Models for Named Entity Typing and Recognition},
  author={Epure et al. (2021)},
  year={2021},
  note={arXiv:2108.11857}
}

• arXiv: 2108.11857