Hallucination Mitigation Eval

hallucination-mitigation-eval

Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs — Jo et al. (2026) (arXiv:2601.13707, 2026)

What this evaluates

Evaluates the ability of Large Vision-Language Models to generate factually aligned outputs by measuring object hallucination rates in captions and yes/no answers, as well as logical reasoning and attribute consistency across diverse visual prompts.

Datasets

• POPE — total ?; splits: random (-1), popular (-1), adversarial (-1), total (-1)
• CHAIR — total ?; splits: test (-1)
• MMHal-Bench — total 96; splits: test (96)

Metrics

• POPE Average Accuracy (primary) — range: percent
  • Accuracy of binary yes/no predictions averaged across the random, popular, and adversarial splits.
• CHAIRs — range: percent
  • Percentage of generated sentences containing at least one hallucinated object.
• CHAIRi — range: percent
  • Percentage of images where the generated caption contains at least one hallucinated object.
• CHAIR F1 — range: percent
  • Harmonic mean of recall and precision for detected objects in captions compared to ground truth.
• MMHal-Bench Average Score — range: other
  • GPT-4 judged alignment score averaged across eight categories (ATTR, ADV, COMP, COUNT, SPAT, ENV, HOL, OTHER).

Input / output format

Input: Image and text prompt (question or instruction) for generation; for POPE, image and yes/no question.

Output: Text response: binary 'yes'/'no' for POPE; free-form caption or answer for CHAIR and MMHal-Bench.

Scoring recipe

def score_poPE(preds, golds):
    accs = [1.0 if p.strip().lower() == g.strip().lower() else 0.0 for p, g in zip(preds, golds)]
    return sum(accs) / len(accs)

def score_chair(preds, gold_objs):
    sents_halluc = 0; imgs_halluc = 0; tp=fp=fn=0
    for cap, gt in zip(preds, gold_objs):
        detected = parse_objects(cap)
        tp += len(detected & gt)
        fp += len(detected - gt)
        fn += len(gt - detected)
        if detected - gt:
            imgs_halluc += 1
            sents_halluc += count_sentences_with_halluc(cap, detected - gt)
    return (sents_halluc/len(preds), imgs_halluc/len(preds), 2*tp/(2*tp+fp+fn))

Common pitfalls

• CHAIR metrics depend heavily on the object detection parser used to extract entities from generated text; inconsistent NLP pipelines yield different CHAIRs/CHAIRi values.
• POPE adversarial split contains semantically or statistically related negative samples, making it significantly harder than random/popular splits and requiring careful prompt formatting.
• MMHal-Bench relies on GPT-4 for scoring, which can introduce judge bias or variability; the paper notes that weighting parameter gamma must be tuned per model for stable behavior.

Evidence (verbatim from paper)

POPE measures binary yes/no object existence, while CHAIR evaluates object hallucinations in free-form captions. MMHal-Bench consists of 96 image–question pairs that probe object and attribute-level inconsistencies. Model responses’ alignment with ground-truth answers is evaluated by GPT-4. We report sentence-level hallucination rate (CHAIRs), instance-level hallucination rate (CHAIRi), and F1 score.

Citation

@misc{jo2026attention,
  title={Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs},
  author={Jo et al. (2026)},
  year={2026},
  note={arXiv:2601.13707}
}

• arXiv: 2601.13707