Tweeteval Eval

tweeteval-eval

TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification — Barbieri et al. (2020) (arXiv:2010.12421, 2020)

What this evaluates

Evaluates the ability of language models to classify short social media posts across seven distinct Twitter-specific tasks, including sentiment, emotion, hate speech, and irony detection. It probes domain adaptation by comparing models pre-trained on generic text versus those further trained on large-scale Twitter corpora.

Datasets

• TweetEval — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/cardiffnlp/tweeteval

Metrics

• M-F1 (primary) — range: [0, 1]
  • Macro-averaged F1 score, computed as the unweighted mean of the F1 scores for each class. Calculated as 2 * (precision * recall) / (precision + recall) per class, then averaged across all classes.
• F(i) — range: [0, 1]
  • F1 score specifically used for the irony detection task.
• M-Rec — range: [0, 1]
  • Macro-averaged recall, computed as the unweighted mean of recall scores across all classes.
• AVG (F(a), F(f)) — range: [0, 1]
  • Average of the F1 scores for the two stance targets (against and for), used for the stance prediction task.

Input / output format

Input: Raw tweet text (strings, typically ≥3 tokens, with URLs removed).

Output: Discrete class label corresponding to the specific classification task (e.g., positive/negative for sentiment, emotion categories, offensive/non-offensive, etc.).

Scoring recipe

def compute_macro_f1(predictions, gold_labels, num_classes):
    class_f1s = []
    for c in range(num_classes):
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        class_f1s.append(f1)
    return sum(class_f1s) / len(class_f1s)

Common pitfalls

• Training data for emoji prediction was downscaled to ≤50k tweets per task due to Twitter's data distribution policy, leading to lower performance compared to original SemEval benchmarks that used larger datasets.
• Validation and test splits for hate speech were collected at different timespans, causing topic distribution shifts and significant performance drops when models are optimized on the validation set.
• Models should be evaluated on the test set using the best hyperparameters selected on the validation set, not the validation scores themselves.

Evidence (verbatim from paper)

Table 3: TweetEval validation and test results. For neural models we report both the average result from three runs and its standard deviation, and the best result according to the validation set (parentheses). SotA results correspond to the best systems in the original shared tasks - they are included for completeness as they not directly comparable. Splits might differ, and * indicates that a larger training set is used.* Metric | | M-F1 | M-F1 | M-F1 | F(i) | M-F1 | M-Rec | AVG (F(a),$F^{(f)}$) | TE |

Citation

@misc{barbieri2020tweeteval,
  title={TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification},
  author={Barbieri et al. (2020)},
  year={2020},
  note={arXiv:2010.12421}
}

• arXiv: 2010.12421