Social Media Bias Eval

social-media-bias-eval

Intertwined Biases Across Social Media Spheres: Unpacking Correlations in Media Bias Dimensions — Liu et al. (2024) (arXiv:2408.15406, 2024)

What this evaluates

This benchmark evaluates the ability of models to automatically detect multiple dimensions of media bias (e.g., hate speech, racial, gender, political, linguistic, and text-level context bias) in social media posts across different topic domains. It probes a model's robustness to domain shift and severe class imbalance in multi-label bias identification tasks.

Datasets

• Social Media Bias Dataset (YouTube & Reddit) — total ?; splits: test (-1)

Metrics

• weighted average F1 score (primary) — range: [0, 1]
  • The F1 score is computed for each class (positive/negative for each bias dimension) and then averaged, weighting each class by its support (number of true instances). This accounts for the severe class imbalance where less than 20% of annotations are positive.

Input / output format

Input: Raw text of social media posts from YouTube and Reddit, categorized into five topic domains (politics, sports, healthcare, job & education, entertainment).

Output: Binary labels (positive/negative) for each of the six bias dimensions: Gender Bias, Racial Bias, Hate Speech, Linguistic Bias, Text-level Context Bias, and Political Bias.

Scoring recipe

def compute_weighted_f1(y_true, y_pred):
    f1_scores = []
    weights = []
    for label in [0, 1]:
        tp = np.sum((y_true == label) & (y_pred == label))
        fp = np.sum((y_true != label) & (y_pred == label))
        fn = np.sum((y_true == label) & (y_pred != label))
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
        f1_scores.append(f1)
        weights.append(np.sum(y_true == label))
    return np.average(f1_scores, weights=weights)

Common pitfalls

• Severe class imbalance (<20% positive annotations) makes accuracy misleading; weighted F1 is required.
• Models trained on external datasets (e.g., MBIB) suffer significant performance drops due to domain shift, making cross-dataset evaluation unreliable without fine-tuning.
• Nuanced definitions of political, linguistic, and text-level context biases lead to inherently lower prediction performance and lower inter-rater agreement.

Evidence (verbatim from paper)

To account for class imbalance, we use weighted average F1 score as our evaluation metrics for all our automated annotations following practices in prior works [13], [33]. For the models trained on MBIB datasets, we adopt a random 5-fold train-validation split with a 75% data used for training and 25% data used for validation, where the best-performing model in validation set is used for evaluation on our collected social media posts.

Citation

@misc{liu2024intertwined,
  title={Intertwined Biases Across Social Media Spheres: Unpacking Correlations in Media Bias Dimensions},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2408.15406}
}

• arXiv: 2408.15406