Videohallucer Eval

videohallucer-eval

VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models — Wang et al. (2024) (arXiv:2406.16338, 2024)

What this evaluates

This benchmark evaluates large video-language models for intrinsic and extrinsic hallucinations by presenting paired basic and adversarially modified Yes/No questions about video content. It probes whether models can correctly identify factual content while resisting fabricated or unverifiable details, and measures susceptibility to language bias.

Datasets

• VideoHallucer — total 1800; splits: test (1800)

Metrics

• Overall Accuracy (primary) — range: percent
  • Percentage of video-question pairs where both the basic question and its paired hallucinated question are answered correctly. A pair counts as a hit only if both answers match the ground truth.
• Yes Percentage Difference — range: percent
  • Absolute difference between the proportion of 'yes' predictions and the proportion of 'yes' ground truths across all pairs: d_y = |M(v,q)='yes'|/|V| - |GT(v,q)='yes'|/|V|.
• False Positive Ratio — range: percent
  • Proportion of 'yes' answers among all incorrectly predicted pairs: r_fp = |M(v,q)='yes'|_{(v,q)∈W} / |W|, where W is the set of wrongly answered pairs.

Input / output format

Input: A video clip (ranging from 7 to 187 seconds) paired with a Yes/No question (either a basic factual question or an adversarially hallucinated variant).

Output: A binary 'Yes' or 'No' response.

Scoring recipe

correct_overall = 0
correct_basic = 0
correct_hall = 0
yes_preds = 0
yes_gt = 0
wrong_pairs = []
for v, q_b, q_h in pairs:
  pred_b = model.predict(v, q_b)
  pred_h = model.predict(v, q_h)
  gt_b = gold(v, q_b)
  gt_h = gold(v, q_h)
  if pred_b == gt_b: correct_basic += 1
  if pred_h == gt_h: correct_hall += 1
  if pred_b == gt_b and pred_h == gt_h: correct_overall += 1
  if pred_b == 'yes': yes_preds += 1
  if gt_b == 'yes': yes_gt += 1
  if pred_b != gt_b or pred_h != gt_h: wrong_pairs.append((v, q_b, q_h))
overall_acc = correct_overall / len(pairs) * 100
pct_diff = abs(yes_preds - yes_gt) / len(pairs) * 100
fp_ratio = sum(1 for _, qb, qh in wrong_pairs if model.predict(v, qb)=='yes' or model.predict(v, qh)=='yes') / len(wrong_pairs) * 100

Common pitfalls

• Evaluating basic and hallucinated questions independently instead of as a paired set; the protocol requires a hit only if both are answered correctly.
• Ignoring bias metrics (Pct. Diff and FP Ratio); high accuracy can mask severe language bias where models over-predict 'yes'.
• Assuming scaling models improves extrinsic hallucination detection; the paper shows parameter/dataset scaling yields limited gains for extrinsic factual hallucinations.

Evidence (verbatim from paper)

To mitigate biases such as the distribution of answers and language bias, we develop VideoHallucer using an adversarial approach*(Tong et al., [2024])*. Specifically, for each evaluation item, we formulate two types of questions: a basic question and a hallucinated question. The basic question assesses the core capabilities of LVLMs, while the hallucinated question includes deliberately hallucinated content. We then calculate the overall accuracy by considering both the basic and hallucinated questions as a paired set, marking it as a hit only if both questions are answered correctly.

Citation

@misc{wang2024videohallucer,
  title={VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2406.16338}
}

• arXiv: 2406.16338