Topic Trend Detection Eval

topic-trend-detection-eval

SemEval-2015 Task 10: Sentiment Analysis in Twitter — Rosenthal et al. (2015) (SemEval-2015 / arXiv:1912.02387, 2015)

What this evaluates

Measures the change in sentiment trend towards a specific topic over time or across datasets, requiring temporal or comparative analysis.

Datasets

• Twitter2015-test — total ?; splits: test (-1)

Metrics

• avgDiff (primary) — range: other
  • Average difference in sentiment scores across time periods or datasets for a given topic. Systems are ranked by this metric.
• avgLevelDiff — range: other
  • Average level difference metric used to rank systems; yields identical rankings to avgDiff.

Input / output format

Input: Tweets associated with a topic, ordered or grouped by time/context.

Output: Trend score or direction for the topic.

Scoring recipe

# avgDiff and avgLevelDiff are provided by the task organizers
# Systems are ranked by avgDiff
rank = sort_by(avgDiff, descending)
# avgLevelDiff yields the same ranking

Common pitfalls

• Only three teams improved over the baseline, indicating high difficulty.
• Trend detection is closely related to topic-level polarity; solving C is a prerequisite for D, and rankings differ from C.

Evidence (verbatim from paper)

The results for subtask D are shown in Table 13. ... Finally, note that avgDiff and avgLevelDiff yielded the same rankings.

Citation

@misc{rosenthal2015semeval,
  title={SemEval-2015 Task 10: Sentiment Analysis in Twitter},
  author={Rosenthal et al. (2015)},
  year={2015},
  note={SemEval-2015 / arXiv:1912.02387}
}

• arXiv: 1912.02387