Semeval2023 Task12 Eval

semeval2023-task12-eval

DN at SemEval-2023 Task 12: Low-Resource Language Text Classification via Multilingual Pretrained Language Model Fine-tuning — Homskiy et al. (2023) (arXiv:2305.02607, 2023)

What this evaluates

Multilingual sentiment classification across low-resource African languages, including zero-shot generalization to unseen languages.

Datasets

• SemEval-2023 Task 12 — total ?; splits: train (-1), validation (-1), test (-1); repo https://github.com/Daniil153/SemEval2023_Task12

Metrics

• F1 (primary) — range: [0, 1]
  • Macro-averaged F1 score (harmonic mean of precision and recall across all sentiment classes).

Input / output format

Input: Raw text string in one of the target African languages.

Output: Predicted sentiment label (e.g., positive, negative, neutral).

Scoring recipe

def compute_f1(predictions, gold):
    precisions, recalls = [], []
    for label in unique_labels:
        tp = sum(p == label and g == label for p, g in zip(predictions, gold))
        fp = sum(p == label and g != label for p, g in zip(predictions, gold))
        fn = sum(p != label and g == label for p, g in zip(predictions, gold))
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        precisions.append(prec)
        recalls.append(rec)
    avg_prec = sum(precisions) / len(precisions)
    avg_rec = sum(recalls) / len(recalls)
    return 2 * (avg_prec * avg_rec) / (avg_prec + avg_rec) if (avg_prec + avg_rec) > 0 else 0.0

Common pitfalls

• The paper uses StratifiedKFold 5-fold cross-validation on the training set and ensembles predictions, rather than a single train/test split.
• Zero-shot tracks (Tigrinya, Oromo) are evaluated separately without fine-tuning on those languages, requiring model selection based on the target metric from other languages.
• Text preprocessing was tested but ultimately discarded as it did not improve F1 scores.

Evidence (verbatim from paper)

For every track except the last two (zero-shot), we employed StratifiedKFold (Pedregosa et al., 2011) with 5 folds to partition the training sample into training and validation sets. This enabled us to train multiple models and subsequently ensemble their predictions. ... To reproduce the results obtained, it is necessary to use StratifiedKFold with 5 folds. Train the model on each training fold. ... Table 1: Results of the DN team in all tracks of the competition ... Our F1

Citation

@misc{homskiy2023dnnatsemeval,
  title={DN at SemEval-2023 Task 12: Low-Resource Language Text Classification via Multilingual Pretrained Language Model Fine-tuning},
  author={Homskiy et al. (2023)},
  year={2023},
  note={arXiv:2305.02607}
}

• arXiv: 2305.02607