Masakhanews Eval

masakhanews-eval

MasakhaNEWS: News Topic Classification for African languages — Adelani et al. (2023) (arXiv:2304.09972, 2023)

What this evaluates

This benchmark evaluates the ability of language models and classical ML algorithms to classify news articles into predefined topics across 16 typologically diverse African languages. It probes multilingual representation quality, script handling, and few-shot/fine-tuning performance in low-resource settings.

Datasets

• MasakhaNEWS — total ?; splits: train (20621), dev (2895), test (5209); repo https://github.com/masakhane-io/masakhane-news

Metrics

• weighted F1-score (primary) — range: [0, 1]
  • The F1-score weighted by class support (number of true instances per class). Calculated as the sum of per-class F1 scores multiplied by their respective support, divided by the total support.

Input / output format

Input: Concatenated news headline and full article text (headline + text) for each instance.

Output: A single topic label from the predefined set: business, entertainment, health, politics, religion, sport, technology.

Scoring recipe

def compute_weighted_f1(preds, gold, classes):
    f1s, supports = [], []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
        supports.append(sum(1 for g in gold if g == c))
    return sum(f * s for f, s in zip(f1s, supports)) / sum(supports)

Common pitfalls

• Models trained on headline-only input significantly underperform compared to headline+text concatenation, especially for classical ML methods.
• Zero-shot averages explicitly exclude the source languages used for training (AVG^src), which can mislead if not accounted for.
• Performance drops sharply for languages using scripts not covered by the model's pretraining vocabulary (e.g., Ge'ez script for Amharic/Tigrinya in Flan-T5).

Evidence (verbatim from paper)

Table 3 shows the result of training several models on TRAIN split and evaluation on the TEST split for each language. ... Evaluation is based on weighted F1-score.

Citation

@misc{adelani2023masakhanews,
  title={MasakhaNEWS: News Topic Classification for African languages},
  author={Adelani et al. (2023)},
  year={2023},
  note={arXiv:2304.09972}
}

• arXiv: 2304.09972