Olid Eval

olid-eval

Predicting the Type and Target of Offensive Posts in Social Media — Zampieri et al. (2019) (arXiv:1902.09666, 2019)

What this evaluates

Evaluates models on a three-level hierarchical schema for offensive language in social media, probing the ability to detect offensiveness, categorize offense type, and identify the target of the offensive content.

Datasets

• Offensive Language Identification Dataset (OLID) — total 10000; splits: train (-1), test (-1)

Metrics

• macro-averaged F1-score (primary) — range: [0, 1]
  • The unweighted mean of the F1-score computed independently for each class, then averaged across all classes. This metric treats all classes equally regardless of their frequency in the dataset.

Input / output format

Input: Raw text of a social media post (tweet).

Output: Predicted class label(s) corresponding to the hierarchical level: Level A (OFF/NOT), Level B (TIN/UNT), or Level C (GRP/IND/OTH).

Scoring recipe

def macro_f1(y_true, y_pred):
    classes = set(y_true)
    f1s = []
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        f1s.append(f1)
    return sum(f1s) / len(f1s)

Common pitfalls

• The dataset is highly imbalanced across classes; using accuracy instead of macro-F1 will misrepresent model performance.
• The 'OTH' (others) target class has very few training instances (395) and is semantically heterogeneous, often causing models to achieve 0 performance on it.
• Tasks are evaluated independently in the reported experiments rather than as a strict hierarchical cascade, which may overestimate real-world pipeline performance.

Evidence (verbatim from paper)

As the label distribution is highly imbalanced (see Table3), we evaluate and we compare the performance of the different models using macro-averaged F1-score. We further report per-class Precision (P), Recall (R), and F1-score (F1), and weighted average.

Citation

@misc{zampieri2019predicting,
  title={Predicting the Type and Target of Offensive Posts in Social Media},
  author={Zampieri et al. (2019)},
  year={2019},
  note={arXiv:1902.09666}
}

• arXiv: 1902.09666