Lvlm Hallucination Eval

lvlm-hallucination-eval

Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs — Lyu et al. (2026) (arXiv:2602.15556, 2026)

What this evaluates

Evaluates Large Vision-Language Models on their ability to generate factually consistent outputs aligned with visual input, specifically measuring the reduction of object hallucinations in open-ended generation while preserving general multimodal reasoning and visual grounding capabilities.

Datasets

• POPE — total ?; splits: test (-1)
• CHAIR — total ?; splits: test (-1)
• HallusionBench — total ?; splits: test (-1)
• AMBER — total ?; splits: test (-1)
• VizWiz — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• LLaVA-Wild — total ?; splits: test (-1)
• MM-Vet — total ?; splits: test (-1)

Metrics

• CHAIR (object hallucination score) (primary) — range: percent
  • Measures the proportion of hallucinated objects in open-ended captions relative to ground truth. Standard recall/precision-based scoring is used per benchmark convention.
• POPE (binary hallucination classification) — range: [0, 1]
  • Accuracy of binary classification distinguishing hallucinated vs. non-hallucinated responses.
• HallusionBench (visual consistency) — range: [0, 1]
  • Accuracy on fine-grained visual consistency questions.
• AMBER (grounded reasoning) — range: [0, 1]
  • Accuracy on visually grounded reasoning and generation tasks.
• General Multimodal Accuracy — range: percent
  • Standard accuracy or score aggregation across VizWiz, MME, LLaVA-Wild, and MM-Vet for visual understanding and reasoning.

Input / output format

Input: Image(s) paired with text instructions/prompts for visual understanding, reasoning, or open-ended captioning tasks.

Output: Text responses generated via sampling-based decoding (default settings).

Scoring recipe

def compute_metrics(predictions, gold):
    scores = {}
    for bench in ['POPE', 'CHAIR', 'HallusionBench', 'AMBER', 'VizWiz', 'MME', 'LLaVA-Wild', 'MM-Vet']:
        if bench == 'CHAIR':
            scores[bench] = calculate_hallucination_recall(predictions[bench], gold[bench])
        elif bench == 'POPE':
            scores[bench] = binary_accuracy(predictions[bench], gold[bench])
        else:
            scores[bench] = standard_accuracy(predictions[bench], gold[bench])
    return scores

Common pitfalls

• Applying intervention at intermediate model layers instead of the final layer, which the ablation shows yields suboptimal performance due to diffuse attention.
• Using excessively large intervention strength (lambda), which introduces strong perturbations that deviate from the model's learned attention dynamics and degrade performance.
• Relying on perturbed visual inputs or external auxiliary models for evaluation, which the paper contrasts with its training-free, internal-dynamics approach.

Evidence (verbatim from paper)

The hallucination-focused benchmarks include POPELi et al. ([2023]) (binary hallucination classification), CHAIRRohrbach et al. ([2018]) (object hallucination in open-ended captioning), HallusionBenchGuan et al. ([2024]) (fine-grained visual consistency), and AMBERWang et al. ([2023]) (visually grounded reasoning and generation).

Citation

@misc{lyu2026pade,
  title={Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs},
  author={Lyu et al. (2026)},
  year={2026},
  note={arXiv:2602.15556}
}

• arXiv: 2602.15556