Vlm Deflection Bench Eval

vlm-deflection-bench-eval

Benchmarking Deflection and Hallucination in Large Vision-Language Models — Moratelli et al. (2026) (arXiv:2604.12033, 2026)

What this evaluates

This benchmark evaluates the ability of large vision-language models to correctly answer knowledge-based visual questions while properly deferring when evidence is missing or hallucinating when faced with noisy or conflicting retrieval contexts. It disentangles parametric memorization from retrieval robustness across four controlled scenarios.

Datasets

• VLM-DeflectionBench — total 2775; splits: test (2775)

Metrics

• Accuracy — range: percent
  • Fraction of responses labeled CORRECT by an external LLM judge (GPT-4o with SimpleQA prompt) against the gold answer.
• Deflection Rate (primary) — range: percent
  • Fraction of responses labeled NOT ATTEMPTED by the judge, indicating the model correctly withheld an answer when evidence was inadequate.
• Hallucination Rate — range: percent
  • Fraction of responses labeled INCORRECT by the judge, indicating the model generated a confident but factually wrong answer.

Input / output format

Input: A question q, an optional image v, and a shuffled set of knowledge contexts K containing one gold passage and multiple distractor passages (text or image).

Output: A short-form factual answer string, or a deferral/withholding response when evidence is inadequate.

Scoring recipe

def evaluate(predictions, gold_answers, judge_model):
    labels = []
    for pred, gold in zip(predictions, gold_answers):
        label = judge_model.evaluate(q, pred, gold)  # Returns CORRECT, INCORRECT, or NOT ATTEMPTED
        labels.append(label)
    accuracy = labels.count("CORRECT") / len(labels)
    deflection = labels.count("NOT ATTEMPTED") / len(labels)
    hallucination = labels.count("INCORRECT") / len(labels)
    return {"Accuracy": accuracy, "Deflection Rate": deflection, "Hallucination Rate": hallucination}

Common pitfalls

• Models frequently generate confident but unsubstantiated claims (hallucinate) instead of deferring when retrieval contexts are noisy or conflicting.
• The benchmark uses a strict RAG setup where contexts are pre-retrieved and shuffled; failing to simulate realistic retrieval noise or ignoring the deferral requirement leads to inflated accuracy scores.
• Evaluation relies on an external LLM judge (GPT-4o) with a specific prompt, which may introduce judge bias or inconsistency compared to exact-match metrics.

Evidence (verbatim from paper)

Responses are judged by an external evaluator $E$: we use GPT-4o as a judge with the SimpleQA promptWei et al. (2024), which assigns one of three labels: $E(q,\hat{a}_{j},a)\in{\texttt{CORRECT},,\texttt{INCORRECT},,\texttt{NOT ATTEMPTED}}222We map these labels to our evaluation metrics as follows: CORRECT $\rightarrow$ Accuracy, INCORRECT $\rightarrow$ Hallucination, NOT ATTEMPTED $\rightarrow$ Deflection.

Citation

@misc{moratelli2026benchmarkingdeflection,
  title={Benchmarking Deflection and Hallucination in Large Vision-Language Models},
  author={Moratelli et al. (2026)},
  year={2026},
  note={arXiv:2604.12033}
}

• arXiv: 2604.12033