Semeval 2014 Task9 Sentiment Eval

semeval-2014-task9-sentiment-eval

SemEval-2014 Task 9: Sentiment Analysis in Twitter — Rosenthal et al. (2014) (SemEval 2014 / arXiv:1912.02990, 2014)

What this evaluates

Evaluates sentiment polarity classification across diverse informal text genres, including tweets, sarcasm-marked tweets, and LiveJournal posts. It distinguishes between phrase-level contextual polarity and message-level sentiment, testing robustness to informal language, sarcasm-induced polarity inversion, and cross-platform generalization.

Datasets

• SemEval-2014 Task 9 Test Sets — total ?; splits: test-twitter (-1), test-twitter-sarcasm (-1), test-livejournal (-1), test-twitter-2013 (-1), test-sms (-1)

Metrics

• macro- and micro-averaged F1 (primary) — range: percent
  • F1 score calculated per polarity class (positive, negative, neutral), then averaged either macro (unweighted mean per class) or micro (weighted by class support). The official ranking uses the Twitter-2014 test set score.

Input / output format

Input: Raw text input (tweet, sarcasm-marked tweet, or LiveJournal sentence)

Output: Categorical polarity label (positive, negative, or neutral)

Scoring recipe

def compute_macro_f1(predictions, gold):
    classes = ['positive', 'negative', 'neutral']
    f1_scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold) if p != cls and g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1_scores.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• Using the 2013 progress test sets for training, which was explicitly forbidden and led to artificially inflated scores.
• Confusing constrained vs. unconstrained system rankings; the official ranking combines both but marks unconstrained runs separately.
• Over-tuning systems on the progress test sets before the official evaluation.

Evidence (verbatim from paper)

The tables further show macro- and micro-averaged results over the 2014 datasets. There is an index for each result showing the relative rank of that result within the respective column. The participating systems are ranked by their score on the Twitter-2014 testset, which is the official ranking for the task; all remaining rankings are secondary.

Citation

@misc{rosenthal2014semeval,
  title={SemEval-2014 Task 9: Sentiment Analysis in Twitter},
  author={Rosenthal et al. (2014)},
  year={2014},
  note={SemEval 2014 / arXiv:1912.02990}
}

• arXiv: 1912.02990