Halluscope Eval

halluscope-eval

When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs — Khayatan et al. (2026) (arXiv:2604.21911, 2026)

What this evaluates

Evaluates LVLMs' ability to resist prompt-induced hallucinations by disentangling perception failures from instruction-induced presuppositions. It probes whether models rely on visual evidence or textual priors when answering questions that imply the presence of non-existent objects.

Datasets

• HalluScope — total ?; splits: test (-1)

Metrics

• AdP (primary) — range: [0, 100]
  • Accuracy on adversarial presupposition tasks, measuring the model's ability to correctly reject false object presuppositions implied in the query. Reported as a percentage.
• Rec_adv — range: [0, 100]
  • Accuracy on adversarial object recognition tasks.
• Rec_pos — range: [0, 100]
  • Accuracy on positive recognition tasks.
• Rec_rand — range: [0, 100]
  • Accuracy on random recognition tasks.

Input / output format

Input: An image paired with a text prompt containing either a direct question, an adversarial presupposition, or a description request.

Output: A natural language response, typically a yes/no answer or a descriptive caption.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for pred, g in zip(predictions, gold) if pred.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100

Common pitfalls

• Metrics are reported as percentages (0-100), not probabilities (0-1).
• AdP specifically measures presupposition rejection, not general hallucination or perception accuracy.
• Baseline comparisons must use greedy decoding for fair comparison, except VCD which uses direct sampling.

Evidence (verbatim from paper)

They demonstrate substantial improvements for HalluVL-DPO in mitigating hallucinations (adversarial recognition and presupposition). For LLaVA, the model becomes remarkably more reliable at identifying objects that are not present when their presence is implied in the query: HalluScope AdP rises from 5.85 to over 80, and CP-Bench CPQ accuracy increases from 0.54 to above 70.

Citation

@misc{khayatan2026halluscope,
  title={When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs},
  author={Khayatan et al. (2026)},
  year={2026},
  note={arXiv:2604.21911}
}

• arXiv: 2604.21911