Tweettopic Eval

tweettopic-eval

Twitter Topic Classification — Antypas et al. (2022) (arXiv:2209.09824, 2022)

What this evaluates

Evaluates language models on classifying tweets into predefined topics, testing both single-label and multi-label classification capabilities. It probes robustness to social media noise, short-form content, and topic overlap in real-world settings.

Datasets

• TweetTopic — total 11267; splits: train (-1), test (-1)

Metrics

• Macro Precision — range: percent
  • Calculated per class as TP/(TP+FP), then averaged across all classes.
• Macro Recall — range: percent
  • Calculated per class as TP/(TP+FN), then averaged across all classes.
• Macro F1 (primary) — range: percent
  • Harmonic mean of macro Precision and macro Recall, averaged across all classes.
• Accuracy — range: percent
  • Proportion of correctly classified tweets out of the total.
• Jaccard Index — range: percent
  • For each tweet, size of intersection over union of predicted and true label sets. Final metric is the average over all tweets.

Input / output format

Input: Raw tweet text.

Output: A single topic label (single-label) or a set of topic labels (multi-label).

Scoring recipe

def compute_macro_f1(preds, golds, classes):
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
    return sum(f1s) / len(f1s)

def compute_jaccard(preds, golds):
    scores = []
    for p, g in zip(preds, golds):
        inter = len(set(p) & set(g))
        union = len(set(p) | set(g))
        scores.append(inter / union if union > 0 else 0)
    return sum(scores) / len(scores)

Common pitfalls

• TimeLM-21 was trained on a corpus overlapping with the test set period, giving it an unfair advantage and inflating its reported performance.
• Multi-label classification is significantly harder than single-label due to label overlap and social media noise, often resulting in much lower F1 scores.
• Macro averaging can mask poor performance on minority topics if not analyzed per-class.

Evidence (verbatim from paper)

For both settings macro average Precision, Recall and F1, as well as Accuracy, are used to evaluate the models tested. As an alternative metric for the multi-label setting, Jaccard Index (JI) is also utilized, as it can offer useful insights about the models performances (Pereira et al., 2018; Tsoumakas et al., 2009). More specifically, the index is calculated for each tweet individually and the final metric is computed as the average over all entries.

Citation

@misc{antypas2022twitter,
  title={Twitter Topic Classification},
  author={Antypas et al. (2022)},
  year={2022},
  note={arXiv:2209.09824}
}

• arXiv: 2209.09824