Universal Ner V2 Eval

universal-ner-v2-eval

Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark — Blevins et al. (2026) (arXiv:2604.12744, 2026)

What this evaluates

Evaluates multilingual named entity recognition (NER) capabilities across 22 languages and 30 datasets, probing both in-language performance and cross-lingual transfer. It also benchmarks large language models as annotators against human inter-annotator agreement to assess guideline adherence and annotation quality.

Datasets

• UNER v2 — total 3000000; splits: train (-1), dev (-1), test (-1)

Metrics

• micro F1 (primary) — range: [0, 1]
  • Standard F1 score computed as 2 * (precision * recall) / (precision + recall), where precision and recall are calculated over predicted versus gold entity spans and their corresponding per/loc/org tags. Micro averaging aggregates true positives, false positives, and false negatives across all datasets and languages before computing the final score.

Input / output format

Input: Sentences from the benchmark datasets, optionally accompanied by the official 3-tag (per/loc/org) annotation guidelines when prompting LLMs.

Output: Predicted entity spans with assigned labels (per, loc, or org) for each input sentence.

Scoring recipe

def compute_f1(predictions, gold):
    pred_spans = set((s, e, l) for s, e, l in predictions)
    gold_spans = set((s, e, l) for s, e, l in gold)
    tp = len(pred_spans & gold_spans)
    fp = len(pred_spans - gold_spans)
    fn = len(gold_spans - pred_spans)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0

Common pitfalls

• Cross-lingual transfer performance drops sharply for typologically distant languages (e.g., Japanese, Korean) compared to European languages.
• LLMs systematically over-annotate non-named entities in some languages (e.g., English) but under-annotate in others (e.g., Hebrew, Korean), failing to consistently follow nuanced guidelines like distinguishing geopolitical entities from locations.
• Entity-level F1 varies significantly by type: 'org' is consistently the hardest to predict, while 'per' is the easiest across most datasets.

Evidence (verbatim from paper)

Figure 3 reports the micro F1 scores on all test sets when XLM-R${}_{\text{Large}}$ is finetuned on different languages. ... Figure 4 presents the inter-annotator agreement between each model and the human annotator who annotated the most documents in a given dataset. ... F1 score comparison across 19 multilingual NER datasets (test sets) for three LLMs against human Inter-Annotator Agreement (IAA) baseline.

Citation

@misc{blevins2026universalner,
  title={Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark},
  author={Blevins et al. (2026)},
  year={2026},
  note={arXiv:2604.12744}
}

• arXiv: 2604.12744