Sold Sentence Eval

sold-sentence-eval

SOLD: Sinhala Offensive Language Dataset — Tharindu Ranasinghe et al. (arXiv:2212.00851, 2022)

What this evaluates

This benchmark evaluates the capability of machine learning models to detect offensive language in Sinhala text. It probes binary text classification performance on a highly imbalanced dataset of Sinhala tweets, measuring how well models distinguish between offensive and non-offensive content.

Datasets

• SOLD — total ?; splits: train (-1), test (-1); repo https://github.com/Sinhala-NLP/SOLD

Metrics

• macro-averaged F1-score (primary) — range: [0, 1]
  • The arithmetic mean of the F1-scores computed for each class (OFF and NOT) independently. It treats all classes equally regardless of their support.

Input / output format

Input: Raw Sinhala sentences (tweets) to be classified.

Output: Binary label indicating whether the input sentence is offensive ('OFF') or not offensive ('NOT').

Scoring recipe

def macro_f1(predictions, gold):
    classes = ['OFF', 'NOT']
    f1_scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold) if p != cls and g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• The dataset has a highly imbalanced label distribution, making accuracy misleading; macro-averaged F1 is required.
• Performance metrics are averaged across five different random seeds to ensure statistical reliability.

Evidence (verbatim from paper)

As the label distribution is highly imbalanced, we evaluate and compare the performance of the different models using macro-averaged F1-score. We further report per-class Precision (P), Recall (R), F1-score (F1), and weighted average.

Citation

@misc{ranasinghe2022sold,
  title={SOLD: Sinhala Offensive Language Dataset},
  author={Tharindu Ranasinghe et al.},
  year={2022},
  note={arXiv:2212.00851}
}

• arXiv: 2212.00851