Pope Eval

pope-eval

A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models — Liqiang Jing et al. (2025) (arXiv:2505.01958, 2025)

What this evaluates

Evaluates object perception and hallucination in LVLMs by prompting models to identify whether specific objects are present in an image. It measures how often models correctly affirm or deny object existence without generating false positives.

Datasets

• POPE — total ?; splits: test (-1)

Metrics

• Acc (primary) — range: [0, 1]
  • Accuracy: proportion of correct predictions out of total instances.
• F1 — range: [0, 1]
  • F1: harmonic mean of precision and recall for the positive class.

Input / output format

Input: Image paired with a yes/no question about object presence.

Output: Yes/No prediction.

Scoring recipe

def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    tp = sum(1 for p, g in zip(preds, golds) if p == g == 'yes')
    fp = sum(1 for p, g in zip(preds, golds) if p == 'yes' and g != 'yes')
    fn = sum(1 for p, g in zip(preds, golds) if p != 'yes' and g == 'yes')
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return acc, f1

Common pitfalls

• Hallucination is measured via binary yes/no accuracy, which may not capture partial matches or nuanced object descriptions.
• The benchmark relies on pre-generated questions, so model performance can be sensitive to question phrasing rather than pure visual understanding.

Evidence (verbatim from paper)

Table 7: Performance of different methods on QA-FB15K.

| Method | Entity | | Relation | | | --- | | | | | | | Acc | F1 | Acc | F1 | | LLaVA-7B | 78.39 | 73.14 | 56.79 | 48.79 | ... Contrastive alignment objective is beneficial for cognition-based knowledge, as evidenced by the performance boost on QA-FB15K.

Citation

@misc{jing2025visualobjecthallucination,
  title={A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models},
  author={Liqiang Jing et al. (2025)},
  year={2025},
  note={arXiv:2505.01958}
}

• arXiv: 2505.01958