# Topic Level Polarity Eval

> Predicts the sentiment polarity associated with a specific topic within a tweet, requiring topic-aware sentiment classification and contextual disambiguation. Use when the user wants to benchmark on Twitter2015-test, or asks about evaluating this task. Reports macro-averaged F1.

- Skill: `qhjqhj00/topic-level-polarity-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/topic-level-polarity-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/topic-level-polarity-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/topic-level-polarity-eval

---


# topic-level-polarity-eval

> SemEval-2015 Task 10: Sentiment Analysis in Twitter — Rosenthal et al. (2015) (SemEval-2015 / arXiv:1912.02387, 2015)

## What this evaluates

Predicts the sentiment polarity associated with a specific topic within a tweet, requiring topic-aware sentiment classification and contextual disambiguation.

## Datasets

- **Twitter2015-test** — total ?; splits: test (-1)

## Metrics

- `macro-averaged F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall averaged across all classes (positive, negative, neutral).

## Input / output format

**Input**: A tweet along with a target topic label.

**Output**: Sentiment polarity label for the specified topic.

## Scoring recipe

```python
prec = rec = 0
for class in ['pos', 'neg', 'neu']:
    prec += precision(y_true, y_pred, class)
    rec += recall(y_true, y_pred, class)
macro_prec = prec / 3
macro_rec = rec / 3
f1 = 2 * (macro_prec * macro_rec) / (macro_prec + macro_rec)
```

## Common pitfalls

- Subtask C is significantly harder than B; top teams score 41-51 F1 vs 64-65 for B, despite similar class distributions.
- Performance drop is due to task complexity (topic-awareness), not class imbalance, as baseline F1 is similar (26.7% vs 30.3%).

## Evidence (verbatim from paper)

> The results for subtask C are shown in Table 12. ... achieved an F1 of 50.51, KLUEless with F1 = 45.48, and Whu-Nlp with F1 = 40.70

## Citation

```bibtex
@misc{rosenthal2015semeval,
  title={SemEval-2015 Task 10: Sentiment Analysis in Twitter},
  author={Rosenthal et al. (2015)},
  year={2015},
  note={SemEval-2015 / arXiv:1912.02387}
}
```

- arXiv: 1912.02387