Contextual Sarcasm Detection Eval

contextual-sarcasm-detection-eval

A Transformer and Prototype-based Interpretable Model for Contextual Sarcasm Detection — Wen et al. (2025) (arXiv:2503.11838, 2025)

What this evaluates

Evaluates a model's ability to detect sarcasm in contextual settings across Reddit comments, tweets, and multi-turn dialogues. It probes the capacity to capture sentiment incongruity and contextual cues rather than relying on surface-level lexical features.

Datasets

• SARC 2.0 — total 1300000; splits: train (118940), test (56118); repo https://nlp.cs.princeton.edu/old/SARC/2.0/
• Twitter — total 1956; splits: train (1368), test (588)
• Sarcasm Corpus V2 Dialogues — total ?; splits: train (-1), test (-1)

Metrics

• accuracy — range: [0, 1]
  • Fraction of correctly classified instances out of the total number of instances.
• recall — range: [0, 1]
  • True positive rate: proportion of actual sarcastic instances correctly identified by the model.
• F1-Score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall. The final reported value is the average across 5-fold cross-validation runs.

Input / output format

Input: Raw text of a social media comment, tweet, or dialogue turn.

Output: Binary classification label indicating whether the text is sarcastic or non-sarcastic.

Scoring recipe

def compute_metrics(preds, gold):
    tp = sum(p == 1 and g == 1 for p, g in zip(preds, gold))
    fp = sum(p == 1 and g == 0 for p, g in zip(preds, gold))
    fn = sum(p == 0 and g == 1 for p, g in zip(preds, gold))
    acc = (tp + sum(p == g for p, g in zip(preds, gold))) / len(gold)
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return acc, rec, f1
# Final reported metric = mean(metric_values across 5 CV folds)

Common pitfalls

• Must use the 'primary main balanced variation' of SARC 2.0, not the raw imbalanced corpus, as class distribution significantly impacts metrics.
• Results must be averaged across 5-fold cross-validation; single-run or single-split scores are not comparable to the reported protocol.
• Prototype distance metric (Euclidean for RoBERTa vs Cosine for SBERT) affects interpretability but does not change the primary classification metric calculation.

Evidence (verbatim from paper)

We used accuracy, recall, and F1-Score as metrics to evaluate models’ performance. Since we use 5-fold cross-validation, we calculated the average of 5 experiment results on the test dataset for each metric as the final result.

Citation

@misc{wen2025transformer,
  title={A Transformer and Prototype-based Interpretable Model for Contextual Sarcasm Detection},
  author={Wen et al. (2025)},
  year={2025},
  note={arXiv:2503.11838}
}

• arXiv: 2503.11838