qhjqhj00 prior-polarity-degree-eval
SemEval-2015 Task 10: Sentiment Analysis in Twitter — Rosenthal et al. (2015) (SemEval-2015 / arXiv:1912.02387, 2015)
What this evaluates
Evaluates the ability of sentiment lexicons or models to assign accurate real-valued polarity scores to individual terms, measuring rank correlation with gold standards.
Datasets
- Term test set — total ?; splits: test (-1)
Metrics
Kendall's τ coefficient(primary) — range: [-1, 1]- Rank correlation coefficient measuring the correspondence between predicted and gold term polarity rankings. Ranges from -1 to 1.
Spearman's ρ coefficient— range: [-1, 1]- Rank correlation coefficient measuring the similarity between predicted and gold term polarity rankings. Ranges from -1 to 1.
Input / output format
Input: A single term (word or phrase).
Output: Real-valued sentiment score.
Scoring recipe
tau = kendalltau(y_true, y_pred)
rho = spearmanrho(y_true, y_pred)
# Higher values indicate better rank correlation with gold scores
Common pitfalls
- Many teams used unsupervised lexicon-based approaches; supervised word-embedding models performed best.
- Baseline uses a simple fallback between two public lexicons (Hashtag Sentiment Lexicon and Sentiment140), scoring ~0.277 avgDiff.
Evidence (verbatim from paper)
The results are presented in Table 14. ... Kendall's τ coefficient, Spearman's ρ coefficient
Citation
@misc{rosenthal2015semeval,
title={SemEval-2015 Task 10: Sentiment Analysis in Twitter},
author={Rosenthal et al. (2015)},
year={2015},
note={SemEval-2015 / arXiv:1912.02387}
}
- arXiv: 1912.02387