Mllm Hallucination Eval

mllm-hallucination-eval

Mitigating Object Hallucinations in MLLMs via Multi-Frequency Perturbations — Li et al. (2025) (arXiv:2503.14895, 2025)

What this evaluates

Evaluates the ability of multimodal large language models (MLLMs) to generate accurate image descriptions and answer questions without hallucinating non-existent objects or attributes. It probes object detection, attribute recognition, and spatial understanding under various prompts.

Datasets

• CHAIR (MSCOCO subset) — total 500; splits: test (500)
• POPE (COCO subset) — total 1500; splits: random (500), popular (500), adversarial (500)
• MME (Hallucination subset) — total ?; splits: test (-1)
• MMBench — total 3000; splits: test (3000)

Metrics

• CHAIR_i — range: [0, 1]
  • Ratio of hallucinated objects to all mentioned objects across generated captions. Lower is better.
• CHAIR_s (primary) — range: [0, 1]
  • Ratio of captions containing at least one hallucinated object to the total number of captions. Lower is better.
• CHAIR F1 — range: [0, 1]
  • F1 score measuring the completeness of generated image descriptions against ground-truth object labels.
• POPE F1 — range: [0, 1]
  • Average F1 score across the random, popular, and adversarial splits for yes/no object probing questions.
• MME Overall — range: other
  • Aggregate score across Existence, Count, Position, and Color subsets evaluating object and attribute hallucinations.
• MMBench Accuracy — range: [0, 1]
  • Percentage of correctly answered multiple-choice questions across 20 ability dimensions.

Input / output format

Input: Image paired with a text prompt. Captioning uses 'Please describe this image in detail.' POPE/MME use targeted questions (e.g., 'Is there a {object} in the image?'). MMBench uses multiple-choice questions.

Output: Free-form text captions for CHAIR, binary/short answers for POPE/MME, and selected option letters for MMBench.

Scoring recipe

def compute_chair_i(captions, gt_objects):
    hallucinated = sum(len(set(c.split()) - set(gt_objects)) for c in captions)
    mentioned = sum(len(set(c.split())) for c in captions)
    return hallucinated / mentioned if mentioned > 0 else 0

def compute_chair_s(captions, gt_objects):
    hallucinated_caps = sum(1 for c in captions if set(c.split()) - set(gt_objects))
    return hallucinated_caps / len(captions)

def compute_pope_f1(preds, labels):
    tp = sum(p == l == 1 for p, l in zip(preds, labels))
    fp = sum(p == 1 and l == 0 for p, l in zip(preds, labels))
    fn = sum(p == 0 and l == 1 for p, l in zip(preds, labels))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• CHAIR scores are lower-better, while POPE F1 and MME scores are higher-better; confusing the directionality leads to incorrect performance claims.
• CHAIR requires exact object grounding against ground-truth labels, not just keyword or semantic matching.
• MME and MMBench are used to verify that hallucination mitigation does not degrade general multimodal capabilities; reporting only CHAIR/POPE gives an incomplete picture.

Evidence (verbatim from paper)

CHAIR consists of two variants: CHAIR_i (instance-level) and CHAIR_s (sentence-level), which are calculated as follows: CHAIR_i = |{hallucinated objects}| / |{all mentioned objects}|, CHAIR_s = |{captions with hallucinated object}| / |{all captions}|. Consistent with the evaluation settings of previous work [29], we randomly sample 500 images from the MSCOCO 2014 validation set [36], set the max-tokens to 512, and use the prompt "Please describe this image in detail." In addition to the CHAIR metric, we also report F1 scores to assess the completeness of the generated image descriptions. ... POPE ... We use the average F1 score of the three split sets as the evaluation metric.

Citation

@misc{li2025mitigating,
  title={Mitigating Object Hallucinations in MLLMs via Multi-Frequency Perturbations},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2503.14895}
}

• arXiv: 2503.14895