Semeval2017task4 Eval

semeval2017task4-eval

SemEval-2017 Task 4: Sentiment Analysis in Twitter — Rosenthal et al. (2017) (arXiv:1912.00741, 2017)

What this evaluates

Evaluates the ability of models to classify sentiment in social media posts (tweets) across different languages (English and Arabic) and granularities (overall polarity, topic-specific polarity, and ordinal scales).

Datasets

• SemEval-2017 Task 4 — total ?; splits: train (-1), test (-1)

Metrics

• macro-average recall (primary) — range: [0, 1]
  • The unweighted mean of recall scores for each sentiment class (Positive, Negative, Neutral). Calculated as the sum of per-class recalls divided by the number of classes.

Input / output format

Input: Tweet text, optionally accompanied by topic labels or user demographic metadata.

Output: A sentiment label (Positive, Negative, Neutral) for classification, or a predicted distribution over sentiment classes for quantification tasks.

Scoring recipe

def macro_avg_recall(y_true, y_pred, classes=['Positive', 'Negative', 'Neutral']):
    recalls = []
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        recalls.append(tp / (tp + fn) if (tp + fn) > 0 else 0.0)
    return sum(recalls) / len(recalls)

Common pitfalls

• Confusing macro-average recall with micro-average recall or accuracy, which can yield different rankings.
• Failing to account for class imbalance when comparing against simple baselines (e.g., predicting all Positive or all Negative).
• Mixing up topic-specific sentiment (Subtasks B-D) with overall tweet sentiment (Subtask A), which requires different model architectures.

Evidence (verbatim from paper)

For English the best ranking teams were BB_twtr and DataStories, both achieving a macro-average recall of 0.681.

Citation

@misc{rosenthal2017semeval,
  title={SemEval-2017 Task 4: Sentiment Analysis in Twitter},
  author={Rosenthal et al. (2017)},
  year={2017},
  note={arXiv:1912.00741}
}

• arXiv: 1912.00741