Graph To Vision Eval

graph-to-vision-eval

Graph-to-Vision: Multi-graph Understanding and Reasoning using Vision-Language Models — Li et al. (2025) (arXiv:2503.21435, 2025)

What this evaluates

This benchmark probes a vision-language model's ability to jointly interpret and reason across multiple related graph images. It specifically tests cross-modal integration, structural understanding, and instruction-following accuracy when processing homogeneous and heterogeneous graph groupings.

Datasets

• Graph-to-Vision Benchmark — total 3459; splits: train (2769), val (390), test (300)

Metrics

• instruction-following accuracy (primary) — range: percent
  • Percentage of model responses that exactly match or semantically align with the human-verified reference answer for each instruction.
• reasoning consistency — range: percent
  • Assessment of whether the model's multi-step reasoning across multiple graphs maintains logical coherence without contradictions.
• graph parsing accuracy — range: percent
  • Accuracy of correctly identifying and extracting graph elements (nodes, edges, structure) from the input images.

Input / output format

Input: A set of interrelated graph images (flowcharts, knowledge graphs, mind maps, or route maps) paired with a natural language instruction requiring cross-graph reasoning.

Output: A natural language response answering the instruction, expected to demonstrate joint interpretation and integration of the provided graph structures.

Scoring recipe

def compute_metrics(predictions, references):
    # Calculate instruction-following accuracy
    exact_matches = sum(1 for p, r in zip(predictions, references) if p.strip() == r.strip())
    accuracy = exact_matches / len(predictions) * 100
    
    # Multi-dimensional framework for consistency and parsing
    # (Paper notes these are assessed via a multi-dimensional framework, likely LLM/human judging)
    consistency_scores = [evaluate_consistency(p) for p in predictions]
    parsing_scores = [evaluate_parsing(p) for p in predictions]
    
    return {
        'instruction_following_accuracy': accuracy,
        'reasoning_consistency': sum(consistency_scores) / len(consistency_scores),
        'graph_parsing_accuracy': sum(parsing_scores) / len(parsing_scores)
    }

Common pitfalls

• Models must handle variable numbers of input images per prompt, which many standard VLMs are not optimized for.
• Heterogeneous graph groupings require cross-type reasoning, which can confuse models trained primarily on single-image or homogeneous multimodal data.
• GPT-4o generated instructions may contain implicit assumptions or biases that require careful human verification to ensure fair evaluation.

Evidence (verbatim from paper)

It evaluates state-of-the-art VLMs via a multi-dimensional framework assessing graph parsing, reasoning consistency, and instruction-following accuracy, and demonstrates consistent performance improvements through fine-tuning on open-source models, while highlighting limitations in scalability due to computational costs.

Citation

@misc{li2025graphtovision,
  title={Graph-to-Vision: Multi-graph Understanding and Reasoning using Vision-Language Models},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2503.21435}
}

• arXiv: 2503.21435