X Topic Eval

x-topic-eval

Multilingual Topic Classification in X: Dataset and Analysis — Antypas et al. (2024) (arXiv:2410.03075, 2024)

What this evaluates

This benchmark evaluates multilingual topic classification on social media tweets across four languages (English, Spanish, Japanese, Greek). It probes models' ability to generalize across languages and training regimes, including zero-shot, few-shot, monolingual, cross-lingual, and multilingual fine-tuning settings.

Datasets

• X-Topic — total 5000; splits: train (3600), val (400), test (1000)

Metrics

• macro-F1 (primary) — range: [0, 1]
  • Computes the unweighted mean of the F1 score for each topic label, assigning equal importance to every class regardless of its frequency in the dataset.
• micro-F1 — range: [0, 1]
  • Aggregates the total true positives, false positives, and false negatives across all topic labels before computing precision and recall, thereby weighting frequent topics more heavily.

Input / output format

Input: A single tweet text in one of four languages (English, Spanish, Japanese, or Greek).

Output: A single topic label from the dataset's fine-grained social media topic taxonomy.

Scoring recipe

def compute_f1(y_true, y_pred, average='macro'):
    # y_true, y_pred: lists of ground-truth and predicted topic labels
    # macro-F1: calculate F1 per class, then take the unweighted mean
    # micro-F1: sum TP, FP, FN globally across all classes, then F1 = 2*P*R/(P+R)
    from sklearn.metrics import f1_score
    return f1_score(y_true, y_pred, average=average)

Common pitfalls

• The few-shot setting uses examples selected from the validation set of each fold (not the training set) to maximize topic coverage.
• The cross-lingual setting trains exclusively on English data but evaluates on other languages, introducing a temporal mismatch between training and test sets.
• Due to significant label imbalance, accuracy is misleading; macro-F1 must be used to fairly evaluate rare topics.

Evidence (verbatim from paper)

Due to the nature of X-Topic, we use the macro-F1 score, which assigns equal weights to each label, as the evaluation metric. This metric is often used for multi-label classification tasks Hazaa et al. ([2023]); Lipton et al. ([2014]); Mohammad et al. ([2018]). In order to better understand the performance of the models and due to the imbalanced nature, which can be a challenge for a model’s performance evaluation He and Garcia ([2009]), micro-F1 is also reported.

Citation

@misc{antypas2024xtopic,
  title={Multilingual Topic Classification in X: Dataset and Analysis},
  author={Antypas et al. (2024)},
  year={2024},
  note={arXiv:2410.03075}
}

• arXiv: 2410.03075