QA Fb15k Eval

qa-fb15k-eval

A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models — Liqiang Jing et al. (2025) (arXiv:2505.01958, 2025)

What this evaluates

Evaluates cognition-based hallucination by testing whether LVLMs can leverage world knowledge stored in the LLM to answer entity and relation questions grounded in images.

Datasets

• QA-FB15K — total ?; splits: test (-1)

Metrics

• Acc (primary) — range: [0, 1]
  • Accuracy: proportion of correct predictions out of total instances.
• F1 — range: [0, 1]
  • F1: harmonic mean of precision and recall for the positive class.

Input / output format

Input: Image paired with a question requiring entity or relation identification based on world knowledge.

Output: Textual answer or predicted label (entity/relation name).

Scoring recipe

def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    tp = sum(1 for p, g in zip(preds, golds) if p == g)
    fp = sum(1 for p, g in zip(preds, golds) if p != g)
    fn = sum(1 for p, g in zip(preds, golds) if p != g)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return acc, f1

Common pitfalls

• Cognition-based benchmarks require both visual grounding and external knowledge; improvements may stem from LLM pretraining rather than vision alignment.
• Entity and Relation splits are evaluated separately, and performance on one does not guarantee generalization to the other due to different knowledge retrieval paths.

Evidence (verbatim from paper)

Table 7: Performance of different methods on QA-FB15K.

| Method | Entity | | Relation | | | --- | | | | | | | Acc | F1 | Acc | F1 | | LLaVA-7B | 78.39 | 73.14 | 56.79 | 48.79 | | Int. Align. 🚀 | 84.28 | 83.03 | 59.16 | 58.07 | | Int. Align. ❄️ | 84.05 | 81.76 | 59.16 | 56.97 | | Sep. Ctrs. Align. | 83.94 | 81.65 | 59.39 | 57.41 | | LLaVA-7B | 78.39 | 73.14 | 56.79 | 48.70 | | w-ECLIP | 77.60 | 71.47 | 56.79 | 45.58 | | w-FineIns | 76.47 | 69.86 | 55.45 | 49.10 |

To further investigate the influence of our method on other kinds of hallucination, we introduced the Cognition-based benchmark: necessitating world knowledge in LVLMs for problem solving. We construct a cognition-based benchmark QA-FB15k based on the knowledge graph FB15K Bordes et al. (2013). We show the results in Table [7].

Contrastive alignment objective is beneficial for cognition-based knowledge, as evidenced by the performance boost on QA-FB15K.

Citation

@misc{jing2025visualobjecthallucination,
  title={A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models},
  author={Liqiang Jing et al. (2025)},
  year={2025},
  note={arXiv:2505.01958}
}

• arXiv: 2505.01958