QA Visualgenome Eval

qa-visualgenome-eval

A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models — Liqiang Jing et al. (2025) (arXiv:2505.01958, 2025)

What this evaluates

Evaluates attribute and relation hallucination by asking LVLMs to identify object properties and inter-object relationships in images, probing fine-grained visual understanding beyond basic object detection.

Datasets

• QA-VisualGenome — total ?; splits: test (-1)

Metrics

• Acc (primary) — range: [0, 1]
  • Accuracy: proportion of correct predictions out of total instances.
• F1 — range: [0, 1]
  • F1: harmonic mean of precision and recall for the positive class.

Input / output format

Input: Image paired with a question about object attributes or relations.

Output: Textual answer or predicted label (attribute/relation name).

Scoring recipe

def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    tp = sum(1 for p, g in zip(preds, golds) if p == g)
    fp = sum(1 for p, g in zip(preds, golds) if p != g)
    fn = sum(1 for p, g in zip(preds, golds) if p != g)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return acc, f1

Common pitfalls

• Attribute and relation hallucinations are conflated with object hallucinations; the benchmark requires strict separation of perception vs. cognition errors.
• Metrics are reported per sub-split (Attribute vs. Relation), so aggregating them without weighting can mask performance drops on specific relation types.

Evidence (verbatim from paper)

Table 7: Performance of different methods on QA-FB15K.

| Method | Entity | | Relation | | | --- | | | | | | | Acc | F1 | Acc | F1 | | LLaVA-7B | 78.39 | 73.14 | 56.79 | 48.79 | ... Contrastive alignment objective is beneficial for cognition-based knowledge, as evidenced by the performance boost on QA-FB15K.

Citation

@misc{jing2025visualobjecthallucination,
  title={A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models},
  author={Liqiang Jing et al. (2025)},
  year={2025},
  note={arXiv:2505.01958}
}

• arXiv: 2505.01958