Cdh Bench Eval

cdh-bench-eval

CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models — Chen et al. (2026) (arXiv:2603.27982, 2026)

What this evaluates

This benchmark probes a vision-language model's ability to maintain visual fidelity when explicit visual evidence conflicts with strong commonsense priors. It specifically measures whether models override entrenched prior-driven expectations with counterfactual image-grounded claims, isolating hallucination from generic perception errors.

Datasets

• CDH-Bench — total ?; splits: test (-1); repo https://github.com/MiLab-HITSZ/2026ChenCDH-Bench

Metrics

• CF-Acc — range: percent
  • Counterfactual Accuracy: the proportion of counterfactual image instances where the model's prediction matches the ground truth.
• CS-Acc — range: percent
  • Commonsense Accuracy: the proportion of matched commonsense control image instances where the model's prediction matches the ground truth.
• CFAD (primary) — range: percent
  • Counterfactual Accuracy Drop: the difference between commonsense accuracy and counterfactual accuracy (CS-Acc - CF-Acc). Lower values indicate less prior-driven collapse.
• CCR — range: percent
  • Commonsense Collapse Rate: the proportion of counterfactual multiple-choice instances where the model outputs the commonsense-consistent distractor instead of the correct counterfactual answer. Reported only for MC tasks.

Input / output format

Input: Paired image-question instances. Each instance provides an image (either a counterfactual variant or a matched commonsense control) along with a text prompt containing a question that embeds a strong commonsense prior and an explicit counterfactual claim. Tasks are formatted as binary QA or multiple-choice QA.

Output: A single predicted answer per instance. For binary QA, a yes/no or true/false response. For multiple-choice QA, a selected option from the provided choices. Answers are extracted deterministically.

Scoring recipe

def compute_metrics(predictions, golds, is_counterfactual):
    cf_acc = sum(1 for p, g, is_cf in zip(predictions, golds, is_counterfactual) if is_cf and p == g) / sum(is_counterfactual)
    cs_acc = sum(1 for p, g, is_cf in zip(predictions, golds, is_counterfactual) if not is_cf and p == g) / sum(not is_cf)
    cfad = cs_acc - cf_acc
    return {'CF-Acc': cf_acc, 'CS-Acc': cs_acc, 'CFAD': cfad}

Common pitfalls

• Interpreting a low CFAD as strong counterfactual grounding without checking CS-Acc, as a model with poor baseline accuracy can mechanically produce a small drop.
• Assuming multiple-choice QA inherently tests superior reasoning, when the benchmark actually shows it amplifies prior-driven failures by pitting counterfactual evidence against plausible commonsense distractors.
• Treating CFAD as a measure of generic perception error; the paired counterfactual-commonsense design specifically isolates prior-driven normalization bias rather than random noise or basic vision failures.

Evidence (verbatim from paper)

Concretely, 7 out of 8 models show lower overall CF-Acc than CS-Acc in both settings; the only partial exception is gemini-3.1-pro-preview, which exhibits a slightly negative CFAD in QA but still shows a 7.33% drop in MC.

Citation

@misc{chen2026cdhbench,
  title={CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2603.27982}
}

• arXiv: 2603.27982