Mbib Political Bias Eval

mbib-political-bias-eval

Navigating Nuance: In Quest for Political Truth — Sar et al. (2025) (arXiv:2501.00782, 2025)

What this evaluates

Evaluates large language models' ability to detect political bias in media text using in-context learning and chain-of-thought prompting. It probes the model's capacity to distinguish biased from unbiased content across diverse textual chunks without fine-tuning.

Datasets

• Media Bias Identification Benchmark (MBIB) — total ?; splits: test (-1)

Metrics

• Macro-F1 (primary) — range: [0, 1]
  • The unweighted mean of the F1 score computed independently for each class (biased vs. unbiased) and then averaged. It treats all classes equally regardless of their support.

Input / output format

Input: Text snippets from the MBIB dataset, formatted with zero-shot, few-shot, or k-shot Chain-of-Thought prompts instructing the model to identify political bias.

Output: A binary label (0 or 1) indicating the presence or absence of political bias. The CoT prompting variant may additionally output brief textual explanations alongside the label.

Scoring recipe

def compute_macro_f1(predictions, gold_labels):
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold_labels) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold_labels) if p == 0 and g == 1)
    
    prec_1 = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec_1 = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1_1 = 2 * prec_1 * rec_1 / (prec_1 + rec_1) if (prec_1 + rec_1) > 0 else 0
    
    prec_0 = (len(predictions) - tp - fp) / (len(predictions) - tp)
    rec_0 = (len(predictions) - tp - fn) / (len(predictions) - fn)
    f1_0 = 2 * prec_0 * rec_0 / (prec_0 + rec_0) if (prec_0 + rec_0) > 0 else 0
    
    return (f1_0 + f1_1) / 2

Common pitfalls

• Few-shot prompting performance heavily depends on example selection, which can mislead the model or fail to instill deeper reasoning.
• CoT prompting may violate strict output instructions by appending explanations, requiring post-processing to extract the binary label for fair comparison.
• Performance is evaluated across 18 distinct chunks rather than a single aggregated score, so reporting must account for chunk-level variance.

Evidence (verbatim from paper)

We also see a nominal increase in the macro-F1 score in the other remaining chunks. ... Note that, the best-performing baseline, i.e. ConvBERT (as reported in*(Wessel et al., [2023])*) achieves an average Macro-F1 score of $0.7110$. We are achieving performance on par with the prompt-based CoT settings when compared to the ConvBERT model...

Citation

@misc{sar2025navigating,
  title={Navigating Nuance: In Quest for Political Truth},
  author={Sar et al. (2025)},
  year={2025},
  note={arXiv:2501.00782}
}

• arXiv: 2501.00782