Semeval 2023 Task12 Eval

semeval-2023-task12-eval

UIO at SemEval-2023 Task 12: Multilingual fine-tuning for sentiment classification in low-resource languages — Rønningstad (2023) (arXiv:2304.14189, 2023)

What this evaluates

Sentiment classification across twelve low-resource African languages and Creoles, evaluating model robustness to code-switching and varying degrees of lexical similarity to pretraining data.

Datasets

• SemEval-2023 Task 12 — total ?; splits: train (-1), test (-1); repo https://github.com/egilron/AfriSenti-SemEval-2023

Metrics

• macro-F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall calculated per sentiment class and averaged across all classes. Standard evaluation metric for SemEval-2023 Task 12.

Input / output format

Input: Raw tweet text (potentially containing code-switching) paired with a ground-truth sentiment label.

Output: Predicted sentiment label for each tweet.

Scoring recipe

def macro_f1(y_true, y_pred, classes):
    f1s = []
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    return sum(f1s) / len(f1s)

Common pitfalls

• Code-switching can disrupt tokenization and degrade performance if the model lacks explicit multilingual or code-switching awareness.
• Class distribution varies significantly across the 12 languages, making macro-averaging essential over micro-averaging to avoid bias toward high-resource languages.
• Multilingual fine-tuning on concatenated datasets often underperforms monolingual fine-tuning due to language interference and pretraining domain mismatch.

Evidence (verbatim from paper)

We trained our model on only the data provided by the shared task. The twelve languages in the training dataset are represented with annotated tweets counting from 804 to 14172 in the training split, as can be seen in Table 2. The dataset by Muhammad et al. (2023a) builds on the work of Muhammad et al. (2022) and Yimam et al. (2020). The dataset includes two Creole languages, Nigerian Pidgin and Mozambican Portuguese, and two arabic languages, Algerian Arabic and Moroccan Arabic / Darija. In addition there is an amount of code-switching in the data (Muhammad et al., 2022). The languages have therefore various levels of similarity, shared vocabulary or closeness to larger languages that our llm was pretrained on. In addition to the training data for each language, the task includes a pre-shuffled dataset containing data from all the individual language datasets, for the multilingual Task B.

Citation

@misc{ronningstad2023uio,
  title={UIO at SemEval-2023 Task 12: Multilingual fine-tuning for sentiment classification in low-resource languages},
  author={Rønningstad (2023)},
  year={2023},
  note={arXiv:2304.14189}
}

• arXiv: 2304.14189