Mbib Eval

mbib-eval

Introducing MBIB -- the first Media Bias Identification Benchmark Task and Dataset Collection — Wessel et al. (2023) (arXiv:2304.13148, 2023)

What this evaluates

This benchmark evaluates a model's ability to identify various forms of media bias, including linguistic, cognitive, political, racial, gender, and hate speech bias, across diverse text sources like news articles, tweets, and social media comments. It probes whether models can generalize across different bias types and dataset sizes without being skewed by larger datasets.

Datasets

• MBIB — total ?; splits: test (-1); repo https://github.com/Media-Bias-Group/MBIB2

Metrics

• micro F1-score — range: [0, 1]
  • Calculated on the predictions of a model on the entire test set by averaging precision and recall over all instances, then computing F1 = 2 * (precision * recall) / (precision + recall).
• macro F1-score (primary) — range: [0, 1]
  • An F1-score is calculated individually for every dataset within a task. The final score is the average of these per-dataset F1-scores, ensuring each dataset is represented equally regardless of size.

Input / output format

Input: Text instances (statements, sentences, tweets, comments, or posts) from diverse sources (e.g., Wikipedia, news articles, Twitter, Reddit) paired with ground-truth bias labels (binary, multi-categorical, or continuous depending on the specific task/dataset).

Output: Predicted bias label(s) for each text instance, matching the label schema of the specific task (e.g., binary bias label, multi-categorical hyperpartisan label, or continuous toxicity/bias score).

Scoring recipe

# Micro F1
all_preds = concat(predictions_from_all_folds)
all_gold = concat(gold_labels_from_all_folds)
micro_f1 = f1_score(all_gold, all_preds, average='micro')

# Macro F1
dataset_f1s = []
for dataset in task_datasets:
    preds_d = get_preds(dataset)
    gold_d = get_gold(dataset)
    dataset_f1s.append(f1_score(gold_d, preds_d, average='macro'))
macro_f1 = mean(dataset_f1s)

Common pitfalls

• Dataset size imbalance: Larger datasets can disproportionately influence the micro F1 score, masking performance on smaller tasks.
• Heterogeneous label spaces: Tasks use different label types (binary, multi-class, continuous), requiring careful handling during aggregation and evaluation.
• Cross-validation vs standard splits: The benchmark uses stratified 5-fold cross-validation rather than a fixed train/val/test split, which affects how scores are reported and compared.

Evidence (verbatim from paper)

As the primary performance metric, we choose the $F_{1}$ -score based on its established usage as a metric in various benchmarks, including those previously discussed [50, 55, 56]. All scores of the five folds are averaged. As the datasets we combine into one task differ in the number of observations they contain, larger datasets can strongly influence the final score. Therefore, to calculate the $F_{1}$ -scores, we propose two methods:

• The micro average $F_{1}$ -score: One $F_{1}$ -score is calculated on the predictions of a model on the entire test set. The scores of the five folds are averaged.
• The macro average $F_{1}$ -score: Multiple $F_{1}$ -scores are calculated on the predictions of a model on the test set. One $F_{1}$ -score is calculated individually for every dataset (from which the data originally stems).

Citation

@misc{wessel2023mbib,
  title={Introducing MBIB -- the first Media Bias Identification Benchmark Task and Dataset Collection},
  author={Wessel et al. (2023)},
  year={2023},
  note={arXiv:2304.13148}
}

• arXiv: 2304.13148