Political Toxicity Annotation Eval

political-toxicity-annotation-eval

Benchmarking LLMs in Political Content Text-Annotation: Proof-of-Concept with Toxicity and Incivility Data — González-Bustamante (2024) (arXiv:2409.09741, 2024)

What this evaluates

Evaluates the ability of LLMs and API-based classifiers to accurately annotate toxicity and incivility in political protest content against a human gold standard. It probes zero-shot classification performance, threshold sensitivity, and output reproducibility across different model sizes and temperatures.

Datasets

• Political protest content dataset — total ?; splits: test (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Used as the headline ranking metric in Table 1.
• accuracy — range: [0, 1]
  • Proportion of correct predictions compared to the human gold standard.
• precision — range: [0, 1]
  • Ability of the classifier to identify positive predicted values and avoid false negatives.
• recall — range: [0, 1]
  • Proportion of correct classifications among true-positive cases.

Input / output format

Input: Raw text messages from political protest interactions.

Output: Binary classification labels (toxic/incivil vs. not) or probability scores (for Perspective API).

Scoring recipe

def compute_metrics(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    accuracy = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    return {'accuracy': accuracy, 'precision': precision, 'recall': recall, 'f1': f1}

Common pitfalls

• Perspective API performance is highly threshold-dependent; using the standard 0.70 cutoff drastically lowers F1-score compared to a 0.55 cutoff, changing its ranking from bottom to top.
• Model reproducibility varies significantly with temperature settings; GPT-4 shows extreme fluctuations at higher temperatures, while open-source models require minimum temperature for full reproducibility.
• Computing time and parameter count do not strictly correlate with performance; smaller models like Nous Hermes 2 and Mistral OpenOrca outperform larger ones in speed and sometimes accuracy.

Evidence (verbatim from paper)

The performance metrics are: (i) accuracy that reports the proportion of correct predictions of the particular classifier in comparison with the human gold standard; (ii) precision that shows the ability of the classifier to identify positive predicted values to identify false negatives; (iii) recall or sensitivity that shows the proportion of correct classifications among true-positive cases; and (iv) F1-score, a combination of precision and recall.

Citation

@misc{gonzalezbustamante2024benchmarking,
  title={Benchmarking LLMs in Political Content Text-Annotation: Proof-of-Concept with Toxicity and Incivility Data},
  author={González-Bustamante (2024)},
  year={2024},
  note={arXiv:2409.09741}
}

• arXiv: 2409.09741