Hpope Eval

hpope-eval

H-POPE: Hierarchical Polling-based Probing Evaluation of Hallucinations in Large Vision-Language Models — Pham et al. (2024) (arXiv:2411.04077, 2024)

What this evaluates

Evaluates hallucinations in large vision-language models by probing their ability to correctly identify object existence and ground fine-grained attributes (color, material, shape) to specific objects in an image.

Datasets

• H-POPE — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Standard binary classification accuracy computed over model responses to yes/no questions about object existence and attributes.

Input / output format

Input: An input image paired with a binary question (e.g., 'Does the [object] have [attribute]?') constructed from ground-truth annotations and negative sampling strategies.

Output: Binary response (Yes/No) from the vision-language model.

Scoring recipe

predictions = [model_answer == 'Yes' for model_answer in model_outputs]
gold = [1 if object/attribute_exists else 0 for _ in questions]
accuracy = sum(p == g for p, g in zip(predictions, gold)) / len(gold)

Common pitfalls

• Models often hallucinate attributes that are present in the image but belong to different objects (image-based adversarial sampling).
• Negative sampling strategies (random, popular, frequency-based, image-based) significantly impact difficulty and should be reported separately.
• Attribute evaluation requires mutual exclusion; models may confuse overlapping or non-mutually exclusive attributes if not restricted to color, material, and shape.

Evidence (verbatim from paper)

Our H-POPE benchmark formulates binary questions about object existence and object attributes. Negative attributes are sampled from the list of attributes across the dataset Ω, so we need to ensure mutual exclusion, i.e., if an object has a list A of attributes, it should not have any attributes from Ω∖A.

Citation

@misc{pham2024hpope,
  title={H-POPE: Hierarchical Polling-based Probing Evaluation of Hallucinations in Large Vision-Language Models},
  author={Pham et al. (2024)},
  year={2024},
  note={arXiv:2411.04077}
}

• arXiv: 2411.04077