Mbib Media Bias Eval

mbib-media-bias-eval

ChatGPT v.s. Media Bias: A Comparative Study of GPT-3.5 and Fine-tuned Language Models — Wen et al. (2024) (arXiv:2403.20158, 2024)

What this evaluates

This evaluation probes a model's ability to detect various forms of media bias in text, including racial, gender, hate speech, fake news, cognitive, and text-level context bias. It tests whether the model can identify subtle, context-dependent linguistic cues and stereotypes across diverse news and social media samples.

Datasets

• Media Bias Identification Benchmark (MBIB) — total ?; splits: train (123123), test (31200)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard binary classification accuracy: the proportion of correctly predicted bias labels (1 for presence, 0 for absence) out of the total test instances.

Input / output format

Input: A text snippet (news article, social media post, or statement) accompanied by a task-specific system prompt instructing the model to identify a specific type of bias (e.g., racial, gender, hate speech).

Output: A JSON object containing a single 'bias' key with a value of 1 (bias present) or 0 (bias absent).

Scoring recipe

correct = 0
total = 0
for pred_json, gold_label in zip(predictions, gold_labels):
    pred_label = pred_json['bias']
    if pred_label == gold_label:
        correct += 1
    total += 1
accuracy = correct / total

Common pitfalls

• Prompt dependency: ChatGPT's performance is highly sensitive to the exact prompt used, which was selected via a small 60-example validation set rather than a rigorous prompt-tuning protocol.
• Label binarization: Continuous labels in the original MBIB are binarized using dataset-specific thresholds recommended by the original authors, which may not align with standard binary classification conventions or the model's natural decision boundary.
• Subsampling bias: Large-scale tasks (Cognitive Bias, Hate Speech) were randomly subsampled to 10% of their original size before the 80/20 split, potentially skewing results compared to full-dataset evaluations.

Evidence (verbatim from paper)

For datasets with continuous labels, binarization is achieved by defining a threshold, with the author's recommended threshold followed wherever possible. ... The performance of these models on the test datasets serves as a comparison against the performance of ChatGPT.

Citation

@misc{wen2024chatgptvsmediabias,
  title={ChatGPT v.s. Media Bias: A Comparative Study of GPT-3.5 and Fine-tuned Language Models},
  author={Wen et al. (2024)},
  year={2024},
  note={arXiv:2403.20158}
}

• arXiv: 2403.20158