Mpd Hallucination Eval

mpd-hallucination-eval

Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation — Zhu et al. (2026) (arXiv:2604.20366, 2026)

What this evaluates

Evaluates the ability of Large Vision-Language Models (LVLMs) to suppress object and semantic hallucinations while maintaining general perception, reasoning, and generative capabilities. It probes grounding fidelity across structured yes/no queries, open-ended captioning, and fine-grained visual diagnostics.

Datasets

• MSCOCO — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• LLaVA-Bench — total ?; splits: test (-1)
• HallusionBench — total ?; splits: test (-1)

Metrics

• CHAIR_S (primary) — range: percent
  • Sentence-level hallucination rate: proportion of generated sentences containing at least one hallucinated object. Lower is better.
• CHAIR_I (primary) — range: percent
  • Instance-level hallucination rate: proportion of generated captions containing at least one hallucinated object. Lower is better.
• POPE F1 (primary) — range: [0, 1]
  • F1 score for yes/no object presence queries under random, popular, and adversarial negative sampling strategies.
• BLEU — range: [0, 1]
  • Standard n-gram overlap metric used to measure caption quality and fluency.
• MME (Existence/Count/Position/Color) — range: other
  • Scores on four hallucination-relevant subsets of the MME benchmark assessing perception and reasoning.
• LLaVA-Bench Accuracy/Detailedness — range: [1, 10]
  • GPT-4V evaluated scores on a 1-10 scale measuring factual alignment and response richness.
• HallusionBench (fACC/qACC/easyA/hardA/aACC) — range: percent
  • Fine-grained accuracy metrics for visual illusion vs language hallucination disentanglement, including easy/hard subsets and overall alignment.

Input / output format

Input: Paired image and text prompt (e.g., open-ended description request or specific yes/no/factual questions).

Output: Free-form natural language response (captions, answers, or descriptions).

Scoring recipe

def score(predictions, gold):
    scores = {}
    scores['CHAIR_S'] = count_hallucinated_sentences(predictions) / len(predictions)
    scores['CHAIR_I'] = count_hallucinated_objects(predictions) / len(predictions)
    scores['POPE_F1'] = f1_score(gold, [pred == 'yes' for pred in predictions])
    scores['BLEU'] = compute_bleu(predictions, gold)
    scores['LLaVA_Acc'] = gpt4v_evaluate(predictions, gold, 'accuracy')
    scores['Hallusion_aACC'] = gpt4v_evaluate(predictions, gold, 'alignment')
    return scores

Common pitfalls

• CHAIR scores represent hallucination rates, so lower values indicate better performance, contrary to most accuracy metrics.
• LLaVA-Bench and HallusionBench rely on GPT-4V for scoring, which can introduce evaluator bias or inconsistency compared to rule-based metrics like POPE.
• MME results only report four hallucination-relevant subsets (Existence, Count, Position, Color), not the full benchmark score.

Evidence (verbatim from paper)

We evaluate OH mitigation performance using the following three categories of benchmarks. MSCOCO-based metrics. We adopt CHAIR and POPE on the MSCOCO dataset to evaluate hallucination in image descriptions. CHAIR includes sentence-level (CHAIR_S) and instance-level (CHAIR_I) hallucination metrics, with lower scores indicating better grounding. POPE measures LVLMs’ ability to correctly answer yes/no object presence queries under different negative sampling strategies.

Citation

@misc{zhu2026mpd,
  title={Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation},
  author={Zhu et al. (2026)},
  year={2026},
  note={arXiv:2604.20366}
}

• arXiv: 2604.20366