Stance Detection Sentiment Eval

stance-detection-sentiment-eval

A Multi-task Model for Sentiment Aided Stance Detection of Climate Change Tweets — Upadhyaya et al. (2022) (arXiv:2211.03533, 2022)

What this evaluates

Evaluates a model's ability to classify stance (in favor, against, neutral) toward climate change and related targets, while jointly predicting sentiment (positive, negative, neutral). It probes the synergy between stance classification and sentiment analysis using text and topic features.

Datasets

• Climate Change Tweets Dataset — total ?; splits: 5-fold CV (-1); repo https://github.com/apoorva-upadhyaya/Climate-Change-Tweets
• SemEval-2016 Task 6.A — total ?; splits: test (-1)

Metrics

• F1 score (primary) — range: percent
  • Harmonic mean of precision and recall, averaged across stance classes (in favor, against, neutral). Reported as mean ± standard deviation over 5 folds.
• Accuracy — range: percent
  • Ratio of correctly predicted instances to total instances. Reported as mean ± standard deviation over 5 folds.

Input / output format

Input: Tweet text concatenated with topic features (5 topics × 10 words each). For SemEval, tweet text with target context.

Output: Two labels per instance: stance label (in favor, against, neutral) and sentiment label (positive, negative, neutral).

Scoring recipe

def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    classes = ['in favor', 'against', 'neutral']
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
    f1 = sum(f1s) / len(f1s)
    return acc * 100, f1 * 100

Common pitfalls

• Oversampling the minority class (deniers) is applied only to the k-1 training folds in each cross-validation split, not the validation fold.
• Topic features are fixed to m=5 topics and p=10 words per topic; varying these changes performance significantly.
• Metrics are reported as mean ± standard deviation over 5 stratified folds, not a single train/test split.

Evidence (verbatim from paper)

For the experiments, we perform stratified k-fold cross-validation on our dataset, oversample the minority class (deniers) in the k-1 training data using the sklearn resampling technique, and report the averaged scores and standard deviation (over 5 folds) for the accuracy and F1 scores.

Citation

@misc{upadhyaya2022multitask,
  title={A Multi-task Model for Sentiment Aided Stance Detection of Climate Change Tweets},
  author={Upadhyaya et al. (2022)},
  year={2022},
  note={arXiv:2211.03533}
}

• arXiv: 2211.03533