Masakhaner Eval

masakhaner-eval

MasakhaNER: Named Entity Recognition for African Languages — Adelani et al. (2021) (arXiv:2103.11811, 2021)

What this evaluates

Evaluates named entity recognition (NER) capabilities across ten African languages, probing models' ability to identify PER, ORG, and LOC entities in low-resource, morphologically complex, and culturally specific news text.

Datasets

• MasakhaNER — total ?; splits: train (-1), test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Standard NER evaluation metric computed over exact entity span matches. Precision = correct entities / predicted entities, Recall = correct entities / gold entities, F1 = 2 * (Precision * Recall) / (Precision + Recall).

Input / output format

Input: Tokenized text sequences (sentences) from African language news articles.

Output: Sequence of BIO/IOB entity labels for each token, indicating PER, ORG, LOC, or O.

Scoring recipe

def compute_f1(predictions, gold):
    pred_spans = extract_spans(predictions)
    gold_spans = extract_spans(gold)
    tp = len(pred_spans & gold_spans)
    fp = len(pred_spans - gold_spans)
    fn = len(gold_spans - pred_spans)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1

Common pitfalls

• Gazetteer augmentation uses English Wikipedia for languages without native Wikipedia (Luo, Nigerian-Pidgin), which may bias results toward English-named entities.
• Cross-lingual transfer experiments restrict evaluation to PER, ORG, and LOC categories because DATE entities are missing from some source domains/languages.
• Domain adaptation (WikiAnn) uses fixed small splits (100 sentences) for most languages but 1K for Swahili, creating uneven transfer baselines.

Evidence (verbatim from paper)

For each language, we train the models on the in-language training data and evaluate on its test data. ... For these experiments, we focus on the PER, ORG, and LOC categories, because the gazetteers from Wikipedia do not contain DATE entities and some source domains and languages that we transfer from do not have the DATE annotation.

Citation

@misc{adelani2021masakhaner,
  title={MasakhaNER: Named Entity Recognition for African Languages},
  author={Adelani et al. (2021)},
  year={2021},
  note={arXiv:2103.11811}
}

• arXiv: 2103.11811