Masakhaner20 Eval

masakhaner20-eval

MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition — Adelani et al. (2022) (arXiv:2210.12391, 2022)

What this evaluates

Evaluates named entity recognition (NER) capabilities across 20 typologically and geographically diverse African languages. It probes zero-shot cross-lingual transfer performance and measures how well models generalize to unseen entities and languages when fine-tuned on limited African language data.

Datasets

• MasakhaNER 2.0 — total ?; splits: train (-1), test (-1)

Metrics

• F1 (primary) — range: percent
  • Standard entity-level F1 score computed over exact match of predicted and gold entity spans and types. Calculated as the harmonic mean of precision and recall.

Input / output format

Input: Tokenized text sequences with corresponding token-level entity annotations (e.g., BIO tags) for each of the 20 target African languages.

Output: Token-level entity labels predicting the type and span of each named entity in the input sequence.

Scoring recipe

def compute_f1(predictions, gold):
    tp = sum(1 for p in predictions if p in gold)
    fp = len(predictions) - tp
    fn = len(gold) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100

Common pitfalls

• Models are evaluated using exact span and type matching; partial overlap or fuzzy matching is not used.
• Results are averaged over 5 independent training runs with different random seeds, not just a single run.
• Zero-shot transfer evaluation requires training on a subset of languages and testing on held-out languages without any fine-tuning on the target language.

Evidence (verbatim from paper)

We fine-tune the PLMs on each language's training data and evaluate performance on the test set using HuggingFace Transformers (Wolf et al., 2020). Table 4 shows the results of training NER models on each language using the eight multilingual and Africa-centric PLMs. ... Average is over 5 runs.

Citation

@misc{adelani2022masakhaner20,
  title={MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition},
  author={Adelani et al. (2022)},
  year={2022},
  note={arXiv:2210.12391}
}

• arXiv: 2210.12391