News Bias Detection Eval

news-bias-detection-eval

Explaining News Bias Detection: A Comparative SHAP Analysis of Transformer Model Decision Mechanisms — Ghosh (2025) (arXiv:2512.23835, 2025)

What this evaluates

Evaluates transformer models' ability to classify news articles as biased or unbiased, comparing standard fine-tuning against domain-adapted training. It further probes model decision-making by analyzing word-level SHAP attribution magnitudes and lexical feature importance across true and false predictions.

Datasets

• BABE — total ?; splits: test (-1)

Metrics

• Binary F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (Precision * Recall) / (Precision + Recall).
• Precision — range: [0, 1]
  • Ratio of correctly predicted positive instances to all instances predicted as positive.
• Recall — range: [0, 1]
  • Ratio of correctly predicted positive instances to all actual positive instances.
• Accuracy — range: [0, 1]
  • Ratio of correct predictions to total predictions.
• False Positive Rate — range: [0, 1]
  • Ratio of false positives to all actual negative instances.
• Macro F1 — range: [0, 1]
  • Unweighted mean of F1 scores for each class.
• Weighted F1 — range: [0, 1]
  • Mean of F1 scores weighted by class support.

Input / output format

Input: News article text (paragraphs or full articles) to be classified for the presence of political/social bias.

Output: Binary label: biased or unbiased (not biased).

Scoring recipe

def compute_metrics(preds, golds):
    tp = sum((p == 1) & (g == 1) for p, g in zip(preds, golds))
    fp = sum((p == 1) & (g == 0) for p, g in zip(preds, golds))
    fn = sum((p == 0) & (g == 1) for p, g in zip(preds, golds))
    tn = sum((p == 0) & (g == 0) for p, g in zip(preds, golds))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    accuracy = (tp + tn) / (tp + fp + fn + tn)
    fpr = fp / (fp + tn) if (fp + tn) > 0 else 0.0
    return {'accuracy': accuracy, 'precision': precision, 'recall': recall, 'binary_f1': f1, 'fpr': fpr}

Common pitfalls

• The paper evaluates two distinct models on the same test set; results must be compared fairly without mixing training data or evaluation splits.
• SHAP attribution magnitudes are analyzed conditionally on prediction correctness (TP vs FP), not just globally; misinterpreting this can lead to wrong conclusions about model alignment.
• Statistical significance is assessed via McNemar's test on the contingency table of errors, not via standard t-tests on metric scores.

Evidence (verbatim from paper)

Table [1] summarizes the performance metrics for both models. While bias-detector achieved higher accuracy and F1, DA-RoBERTa-BABE-FT showed better precision. The significant difference in false positive rates (15.6% vs. 5.7%) suggests that architectural differences and training approaches substantially affect precision in bias detection tasks.

Citation

@misc{ghosh2025explainingnewsbias,
  title={Explaining News Bias Detection: A Comparative SHAP Analysis of Transformer Model Decision Mechanisms},
  author={Ghosh (2025)},
  year={2025},
  note={arXiv:2512.23835}
}

• arXiv: 2512.23835