Openner 1.0 Eval

openner-1.0-eval

OpenNER 1.0: Standardized Open-Access Named Entity Recognition Datasets in 50+ Languages — Palen-Michel et al. (2024) (arXiv:2412.09587, 2024)

What this evaluates

Evaluates named entity recognition (NER) capabilities across 52 languages and 36 distinct corpora. It probes cross-lingual generalization, robustness to varying entity type ontologies, and the ability of both encoder-based models and LLMs to handle multilingual text with diverse annotation guidelines.

Datasets

• OpenNER 1.0 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• micro-averaged mention-level F1 (primary) — range: [0, 1]
  • Standard F1 score computed at the mention level, micro-averaged across all entity types and languages. Precision and recall are calculated by matching predicted entity spans and labels against gold annotations, following the SeqScore/conlleval convention.

Input / output format

Input: Raw text sentences or paragraphs in one of 52 languages, provided as training data for fine-tuning or as prompts for LLM inference.

Output: For fine-tuned models: sequence of BIO/IOB-style entity tags per token. For LLMs: inline labeled text using the format TYPE @@ [Entity Text] ## (e.g., PER @@ John Smith ##).

Scoring recipe

def compute_f1(predictions, gold):
    tp, fp, fn = 0, 0, 0
    for pred, gold_set in zip(predictions, gold):
        if pred == gold_set: tp += 1
        elif pred != gold_set: fp += 1
        if gold_set not in predictions: fn += 1
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return micro_average(f1_across_datasets)

Common pitfalls

• LLM hallucinated tokens must be carefully discarded or penalized as false positives during scoring.
• Generated labels may not match the target ontology, requiring strict mapping or rejection before evaluation.
• Entity type ontologies vary across the 36 datasets; evaluating on 'core types' (PER, ORG, LOC) vs 'full ontologies' yields significantly different results.
• Training data imbalance is mitigated by capping datasets at 32k examples per language, which affects baseline comparability.

Evidence (verbatim from paper)

We report micro-averaged mention-level F1 for each model computed with SeqScore using the same method as the conlleval script. We report the mean and standard error of the mean over 10 different training runs, each using a different random seem for initialization.

Citation

@misc{palenmichel2024openner,
  title={OpenNER 1.0: Standardized Open-Access Named Entity Recognition Datasets in 50+ Languages},
  author={Palen-Michel et al. (2024)},
  year={2024},
  note={arXiv:2412.09587}
}

• arXiv: 2412.09587