Kg Vip Eval

kg-vip-eval

KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering — Li et al. (2026) (arXiv:2601.11632, 2026)

What this evaluates

Evaluates multi-modal LLMs' ability to perform visual question answering by grounding visual inputs with external knowledge graphs. It probes the model's capacity for multi-hop reasoning, visual perception, and knowledge retrieval-augmented generation.

Datasets

• FVQA 2.0+ — total 2820; splits: test (2820)
• MVQA — total 1433; splits: test (1433)

Metrics

• LLM-J (primary) — range: percent
  • LLM-as-a-Judge score where an LLM assigns a scalar score evaluating the relevance of the generated response with the ground-truth answer. Default judge is DeepSeek-V3.2.
• METEOR — range: percent
  • Harmonic mean of unigram precision and recall, capturing semantic variations while applying a fragmentation penalty for word order.
• SAS — range: percent
  • Semantic Answer Similarity score that utilizes a cross-encoder to assess the semantic equivalence between predicted and reference answers.

Input / output format

Input: A query image and a natural language question requiring external knowledge.

Output: A natural language answer string.

Scoring recipe

def compute_metrics(predictions, gold_answers):
    meteor_scores = [meteor_score(pred, gold) for pred, gold in zip(predictions, gold_answers)]
    sas_scores = [cross_encoder_similarity(pred, gold) for pred, gold in zip(predictions, gold_answers)]
    llm_j_scores = [llm_judge_score(pred, gold) for pred, gold in zip(predictions, gold_answers)]
    return {
        'METEOR': mean(meteor_scores) * 100,
        'SAS': mean(sas_scores) * 100,
        'LLM-J': mean(llm_j_scores) * 100
    }

Common pitfalls

• FVQA 2.0+ is a custom multimodal upgrade of the original FVQA dataset with a visual commonsense graph, not the standard textual FVQA benchmark.
• LLM-J scores are highly sensitive to the specific judge model (DeepSeek-V3.2 vs Qwen2.5-7B) and prompting strategy, which are not fully detailed in the main text.
• MVQA is a newly constructed, non-public dataset based on MovieBench, so external reproducibility requires following the Appendix A.2 pipeline.

Evidence (verbatim from paper)

Following prior work, we adopt three widely used metrics for evaluation. METEOR calculates the harmonic mean of unigram precision and recall, capturing semantic variations while applying a fragmentation penalty for word order. Semantic Answer Similarity (SAS) utilizes a cross-encoder to assess the semantic equivalence between predicted and reference answers. LLM-as-a-Judge leverages an LLM to assign a scalar score that evaluates the relevance of the generated response with the ground-truth answer. We denote this metric as LLM-J for short.

Citation

@misc{li2026kgvip,
  title={KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question Answering},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2601.11632}
}

• arXiv: 2601.11632