Besstie Eval

besstie-eval

BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of English — Srirag et al. (2024) (arXiv:2412.04726, 2024)

What this evaluates

Evaluates language models' ability to classify sentiment and detect sarcasm across three distinct varieties of English (Australian, Indian, and British). It probes cross-variety generalization and the impact of domain (Google reviews vs. Reddit comments) on model performance.

Datasets

• BESSTIE — total ?; splits: train (-1), test (-1)

Metrics

• F-Score (primary) — range: [0, 1]
  • Macro-averaged F1-score, computed as the harmonic mean of precision and recall across all classes. Values range from 0 to 1, with higher indicating better performance.

Input / output format

Input: Textual user-generated content (Google Places reviews or Reddit comments) labeled with a specific English variety (en-AU, en-IN, or en-UK).

Output: Classification labels for sentiment (e.g., positive/negative) and sarcasm (e.g., sarcastic/not sarcastic).

Scoring recipe

def compute_f1(predictions, gold):
    # Calculate precision and recall per class
    # Macro-average across classes
    # Return F1 score
    pass

Common pitfalls

• Sarcasm labels are only available for the Reddit subset; Google Places reviews lack sarcasm annotations.
• Fine-tuning on a single variety significantly degrades cross-variety generalization for sarcasm classification.
• Encoder models consistently outperform decoder models on these classification tasks due to architectural suitability.

Evidence (verbatim from paper)

We first present our results on the two tasks where models are trained and evaluated on the same variety. Figures[3] and[4] describe the model performances, reported using F-Score, on the sentiment and sarcasm classification tasks respectively. While sentiment classification is performed using both google and reddit subsets, due to the absence of sarcasm labels in google subset (as shown in Table[5]), sarcasm classification is conducted only on reddit subset.

Citation

@misc{srirag2024besstie,
  title={BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of English},
  author={Srirag et al. (2024)},
  year={2024},
  note={arXiv:2412.04726}
}

• arXiv: 2412.04726