Offenseval Eval

offenseval-eval

SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval) — Zampieri et al. (2019) (arXiv:1903.08983, 2019)

What this evaluates

Evaluates models' ability to detect offensive language in social media posts, classify the specific type of offense (e.g., insult, threat), and identify the target of the offense (individual vs. group). It tests fine-grained text classification and hierarchical annotation understanding in noisy, short-form text.

Datasets

• OLID — total 14000; splits: test (-1)

Metrics

• F1-Macro (primary) — range: [0, 1]
  • Macro-averaged F1 score computed independently for each class and then averaged across all classes. Calculated as the harmonic mean of precision and recall for each class, with equal weight given to all classes regardless of frequency.

Input / output format

Input: Raw social media tweets (text strings)

Output: Sub-task A: binary label (offensive / not offensive). Sub-task B: offense type label (e.g., insult, threat). Sub-task C: target label (individual / group / none).

Scoring recipe

def compute_macro_f1(predictions, gold):
    classes = set(gold) | set(predictions)
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• Sub-tasks A, B, and C are evaluated independently; high performance on one does not guarantee performance on others.
• Macro-F1 is used, which treats all classes equally and can mask poor performance on minority classes due to dataset imbalance.
• Teams applied highly variable pre-processing steps (e.g., hashtag segmentation, emoji substitution, token normalization), making direct model comparisons difficult without standardized pipelines.

Evidence (verbatim from paper)

The results for each of the sub-tasks are shown in Table 4. Due to the large number of submissions, we only show the F1-score for the top-10 teams, followed by result ranges for the rest of the teams. ... Table 4: F1-Macro for the top-10 teams followed by the rest of the teams grouped in ranges for all three sub-tasks.

Citation

@misc{zampieri2019offenseval,
  title={SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)},
  author={Zampieri et al. (2019)},
  year={2019},
  note={arXiv:1903.08983}
}

• arXiv: 1903.08983