Conll Ner Eval

conll-ner-eval

POLYGLOT-NER: Massive Multilingual Named Entity Recognition — Al-Rfou et al. (2014) (arXiv:1410.3791, 2014)

What this evaluates

Evaluates a model's ability to perform Named Entity Recognition (NER) across multiple languages, specifically testing its robustness to out-of-domain text, orthographic variations, and cross-lingual transfer when trained on noisy Wikipedia-derived data.

Datasets

• CoNLL 2002/2003 NER — total ?; splits: dev (-1), test (-1)

Metrics

• Exact F1 (primary) — range: [0, 1]
  • Exact match F1 score computed over token-level entity spans and labels. A prediction is correct only if the predicted span exactly matches the gold span and the entity type (Person, Location, Organization) matches exactly.

Input / output format

Input: Raw text sentences or paragraphs in various languages (English, Spanish, Dutch).

Output: Token-level sequence labels indicating entity types (e.g., PERSON, LOCATION, ORGANIZATION) using a standard tagging scheme like BIO.

Scoring recipe

def exact_f1(preds, golds):
    pred_spans = extract_spans(preds)
    gold_spans = extract_spans(golds)
    tp = len([s for s in pred_spans if s in gold_spans])
    fp = len(pred_spans) - tp
    fn = len(gold_spans) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Out-of-domain evaluation: Models are trained on Wikipedia data but tested on CoNLL, causing performance drops due to stylistic and orthographic differences.
• Lack of dataset-specific preprocessing: The system does not apply CoNLL-tailored normalization (e.g., handling trailing periods, abbreviations like 'Spa' for Spain), leading to higher OOV rates and missed entities.
• Label misclassification: Confusion between nested entities (e.g., LOCATION vs ORGANIZATION) and demonyms/nationalities.

Evidence (verbatim from paper)

Table 6: Cross-domain performance measured by Exact $F_{1}$ on TEST and DEV sections of CONLL corpora. In addition to the qualitative analysis, we evaluate our models quantitatively on the CoNLL 2002 Spanish and Dutch datasets, and the CoNLL 2003 English dataset. We show results of our models trained on Wikipedia and evaluated on CoNLL in Table 6.

Citation

@misc{alrfou2014polyglotner,
  title={POLYGLOT-NER: Massive Multilingual Named Entity Recognition},
  author={Al-Rfou et al. (2014)},
  year={2014},
  note={arXiv:1410.3791}
}

• arXiv: 1410.3791