Vilbias Eval

vilbias-eval

ViLBias: Detecting and Reasoning about Bias in Multimodal Content — Raza et al. (2024) (arXiv:2412.17052, 2024)

What this evaluates

This benchmark probes a model's ability to detect framing bias in multimodal news content (text-image pairs) and generate grounded, correct rationales for its decisions. It evaluates both closed-ended classification accuracy and open-ended reasoning quality using an LLM-as-judge protocol.

Datasets

• ViLBias — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/shainarazavi/VILBias

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly classified text-image pairs (biased vs. not biased) against ground-truth annotations.
• F1 Score — range: percent
  • Harmonic mean of precision and recall for the biased class, reported as a percentage.
• Reasoning Accuracy — range: percent
  • LLM-judged percentage of rationales that correctly support the ground-truth classification decision.
• Faithfulness — range: percent
  • LLM-judged percentage of rationales that are fully supported by the provided text and image evidence.

Input / output format

Input: Text-image pairs representing news media content, optionally accompanied by a prompt requesting a bias label and/or a rationale.

Output: Closed-ended: categorical label ('biased' or 'not biased'). Open-ended: categorical label followed by a concise rationale explaining the decision.

Scoring recipe

def compute_classification_metrics(preds, golds):
    tp = sum(1 for p, g in zip(preds, golds) if p == 'biased' and g == 'biased')
    fp = sum(1 for p, g in zip(preds, golds) if p == 'biased' and g == 'not biased')
    fn = sum(1 for p, g in zip(preds, golds) if p == 'not biased' and g == 'biased')
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    accuracy = sum(1 for p, g in zip(preds, golds) if p == g) / len(golds)
    return accuracy, f1

def compute_reasoning_metrics(rationales, golds, inputs, judge_model='GPT-4o'):
    reasoning_acc = sum(1 for r, g in zip(rationales, golds) if judge_model(f'Does rationale {r} correctly support ground truth {g}?')) / len(golds)
    faithfulness = sum(1 for r, i in zip(rationales, inputs) if judge_model(f'Is rationale {r} fully supported by input {i}?')) / len(golds)
    return reasoning_acc, faithfulness

Common pitfalls

• LLM-as-judge bias: Using GPT-4o to evaluate rationales may favor models with similar prompting styles or over-penalize stylistic differences rather than factual grounding.
• Open-ended vs closed-ended gap: Reasoning accuracy consistently lags classification accuracy by 6–12 points; evaluating only closed-ended metrics overestimates a model's true reasoning capability.
• Cross-validation vs fixed splits: The paper uses both an 80/10/10 split and 5-fold CV; results may vary depending on which protocol is reported or averaged.

Evidence (verbatim from paper)

Protocols and metrics. We evaluated models under three regimes: zero-shot, five-shot, and full fine-tuning. The dataset was partitioned into training, validation, and test splits with an 80/10/10 ratio, and we additionally employed 5-fold cross-validation to assess robustness. For classification tasks, we report standard metrics including precision, recall, F1 score, and accuracy. To capture performance on open-ended outputs, we further evaluate reasoning quality using LLM-as-judge assessments, focusing on reasoning accuracy and faithfulness. We used GPT4o as the judge model.

Citation

@misc{raza2024vilbias,
  title={ViLBias: Detecting and Reasoning about Bias in Multimodal Content},
  author={Raza et al. (2024)},
  year={2024},
  note={arXiv:2412.17052}
}

• arXiv: 2412.17052