Afrisenti Eval

afrisenti-eval

AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages — Shamsuddeen Hassan Muhammad et al. (2023) (arXiv:2302.08956, 2023)

What this evaluates

Evaluates multilingual and cross-lingual sentiment classification capabilities on low-resource African languages using Twitter data. It probes how well pre-trained language models handle dialectal variation, code-switching, and mixed scripts in fine-tuning and zero-shot transfer settings.

Datasets

• AfriSenti — total ?; splits: train (-1), test (-1); repo https://github.com/afrisenti-semeval/afrisent-semeval-2023

Metrics

• F1 (primary) — range: percent
  • Macro-averaged F1 score across sentiment classes. Computed as the harmonic mean of precision and recall per class, then averaged across all classes.

Input / output format

Input: Raw tweet text in one of 14 African languages.

Output: Predicted sentiment label (positive, negative, or neutral).

Scoring recipe

def compute_f1(predictions, gold):
    classes = sorted(set(predictions + gold))
    f1_scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold) if p != cls and g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores) * 100

Common pitfalls

• Zero-shot transfer is only evaluated on Oromo (orm) and Tigrinya (tir), so results do not generalize to other low-resource African languages.
• F1 scores are reported as percentages in tables (e.g., 71.2), but internal calculations may use a 0-1 scale; ensure consistent scaling when reproducing.
• Models are fine-tuned for exactly 10 epochs with fixed learning rates (2e-5 or 1e-5); deviating from this schedule will break reproducibility.

Evidence (verbatim from paper)

We used a standard configuration for text classification fine-tuning on HuggingFace with a learning rate of $2e-5$ for smaller PLMs and $1e-5$ for larger PLMs, a batch size of 128, and 10 epochs. Table 7 shows the results of the monolingual baseline models on AfriSenti. XLM-T performs particularly well on Arabic and Portuguese dialects, i.e., arq, ary and pt-MZ, where it outperforms AfriBERTa by 21.8, 14.2, and 13.0 and AfroXLMR-base by 4.0, 5.9, and 4.7 F1 points respectively.

Citation

@misc{shamsuddeen2023afrisenti,
  title={AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages},
  author={Shamsuddeen Hassan Muhammad et al. (2023)},
  year={2023},
  note={arXiv:2302.08956}
}

• arXiv: 2302.08956