Dialectal Sentiment Classification Eval

dialectal-sentiment-classification-eval

Experiences from Creating a Benchmark for Sentiment Classification for Varieties of English — Srirag et al. (2024) (arXiv:2410.11216, 2024)

What this evaluates

Evaluates sentiment classification performance across different English dialects (en-US, en-AU, en-UK, en-IN) and tests how label proximity, review length, and sentiment density affect model generalization.

Datasets

• Google Place Reviews (Dialectal Sentiment) — total ?; splits: en-US (-1), en-AU (-1), en-UK (-1), en-IN (-1)

Metrics

• F1-Score (primary) — range: [0, 1]
  • Macro-averaged F1-score computed across all sentiment classes (1★–5★). Calculated as the harmonic mean of precision and recall per class, then averaged.

Input / output format

Input: Plain text of user-generated Google Place reviews.

Output: Predicted sentiment label (1★–5★ star rating or mapped simple/hard condition).

Scoring recipe

def compute_f1(predictions, gold):
    classes = [1, 2, 3, 4, 5]
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• 'Simple' vs 'hard' conditions refer to label proximity (e.g., star rating similarity), not linguistic difficulty.
• Sampling strategies (length/density) create non-i.i.d. subsets that may not reflect real-world dialectal distribution.
• Averaging F1 across locales can mask severe performance drops on outer-circle dialects like en-IN.

Evidence (verbatim from paper)

The highest performance on the task is reported by models fine-tuned on en-AU (simple) with the F1-Score of 97.2. The worst performance is reported by the baseline models, with the average F1-Score of 93.5. Models report a degraded performance across reviews from all locales when the labels are changed from simple to hard, with an average decrease in the F1-score of 13.7.

Citation

@misc{srirag2024experiences,
  title={Experiences from Creating a Benchmark for Sentiment Classification for Varieties of English},
  author={Srirag et al. (2024)},
  year={2024},
  note={arXiv:2410.11216}
}

• arXiv: 2410.11216