Phrase Level Polarity Eval

phrase-level-polarity-eval

SemEval-2015 Task 10: Sentiment Analysis in Twitter — Rosenthal et al. (2015) (SemEval-2015 / arXiv:1912.02387, 2015)

What this evaluates

Classifies the sentiment polarity of specific phrases within tweets, requiring models to handle contextual disambiguation, sarcasm, and informal language.

Datasets

• Twitter2015-test — total ?; splits: test (-1)

Metrics

• macro-averaged F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall averaged across all classes (positive, negative, neutral). F1 = 2 * (macro_prec * macro_rec) / (macro_prec + macro_rec).

Input / output format

Input: A phrase extracted from a tweet.

Output: Sentiment polarity label (positive, negative, or neutral).

Scoring recipe

prec = rec = 0
for class in ['pos', 'neg', 'neu']:
    prec += precision(y_true, y_pred, class)
    rec += recall(y_true, y_pred, class)
macro_prec = prec / 3
macro_rec = rec / 3
f1 = 2 * (macro_prec * macro_rec) / (macro_prec + macro_rec)

Common pitfalls

• Class imbalance is significant; the majority class baseline achieves ~38% F1, so models must significantly outperform simple majority voting.
• Sarcasm and slang can flip polarity, requiring contextual models rather than simple bag-of-words approaches.

Evidence (verbatim from paper)

The results (macro-averaged F1 score) for subtask A are shown in Table 10. The participating systems are ranked by their score on the Twitter2015-test dataset, which is the official ranking for subtask A

Citation

@misc{rosenthal2015semeval,
  title={SemEval-2015 Task 10: Sentiment Analysis in Twitter},
  author={Rosenthal et al. (2015)},
  year={2015},
  note={SemEval-2015 / arXiv:1912.02387}
}

• arXiv: 1912.02387