Stance Detection Eval

stance-detection-eval

Sarcasm Detection as a Catalyst: Improving Stance Detection with Cross-Target Capabilities — Nkhata et al. (2025) (arXiv:2503.03787, 2025)

What this evaluates

This benchmark evaluates a model's ability to classify the stance (InFavor, Against, or None) of text towards a specific target or query. It specifically probes cross-target generalization by training on multiple targets and testing on a held-out target, while also assessing robustness to sarcastic or figurative language through intermediate sarcasm pre-training.

Datasets

• SemEval 2016 Task 6A Dataset — total ?; splits: train (-1), test (-1)
• Multi-Perspective Consumer Health Query Data (MPCHI) — total ?; splits: train (-1), test (-1)

Metrics

• average macro F1-score (primary) — range: [0, 1]
  • Macro F1-score computed exclusively over the 'InFavor' and 'Against' classes, then averaged. The 'None' class is explicitly excluded from the calculation.

Input / output format

Input: Text instance (tweet or sentence) paired with a target entity or query. Text is preprocessed via case folding, stemming, stop-word removal, and hashtag processing, then tokenized using the pre-trained language model's default tokenizer.

Output: A single class label from {InFavor, Against, None} produced via a softmax layer over three output nodes.

Scoring recipe

def macro_f1_infavor_against(preds, gold):
    mask = [g != 'None' for g in gold]
    p_f = [x for x, m in zip(preds, mask) if m]
    g_f = [x for x, m in zip(gold, mask) if m]
    f1s = []
    for cls in ['InFavor', 'Against']:
        tp = sum(1 for a, b in zip(p_f, g_f) if a == cls and b == cls)
        fp = sum(1 for a, b in zip(p_f, g_f) if a == cls and b != cls)
        fn = sum(1 for a, b in zip(p_f, g_f) if a != cls and b == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
    return sum(f1s) / len(f1s)

Common pitfalls

• The evaluation explicitly excludes the 'None' class from the F1 calculation; including it will produce incorrect scores.
• Cross-Target Stance Detection (CTSD) uses a leave-one-out split (train on 4 targets, test on 1), not a standard random train/test split.
• Sarcasm detection is used only as an intermediate pre-training task to improve representation learning, not as a direct evaluation metric or output class.

Evidence (verbatim from paper)

In alignment with previous studies[[5]][[7]][[60]], the evaluation of our model is based on the average macro F1-score for the InFavor and Against classes.

Citation

@misc{nkhata2025sarcasm,
  title={Sarcasm Detection as a Catalyst: Improving Stance Detection with Cross-Target Capabilities},
  author={Nkhata et al. (2025)},
  year={2025},
  note={arXiv:2503.03787}
}

• arXiv: 2503.03787