Seam Eval

seam-eval

SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models — Tang et al. (2025) (arXiv:2508.18179, 2025)

What this evaluates

Evaluates vision-language models on their ability to reason across semantically equivalent inputs presented in different modalities (vision vs. language) across domain-specific notation systems. It probes cross-modal consistency, modality-agnostic reasoning, and identifies domain-specific perception and tokenization failure modes.

Datasets

• SEAM — total ?; splits: test (-1); repo https://github.com/CSSLab/SEAM

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly answered questions out of the total number of samples.
• agreement_rate — range: [0, 1]
  • Binary agreement between vision and language answers for each sample (1 if identical, 0 otherwise), averaged across all samples.

Input / output format

Input: Zero-shot chain-of-thought prompt containing a question with domain-specific notation (chess FEN, chemistry SMILES, music ABC, or graph adjacency matrix) presented either as plain text or as an image.

Output: Free-text final answer extracted from the model's generation, typically a single letter/option or short string, post-processed via an external LLM (Qwen2.5-7B-Instruct) using the OpenCompass protocol when rule-based extraction fails.

Scoring recipe

def compute_metrics(preds_vision, preds_language, gold):
    accuracy = sum(1 for p in preds_vision if p == gold) / len(gold)
    agreements = [1 if v == l else 0 for v, l in zip(preds_vision, preds_language)]
    agreement_rate = sum(agreements) / len(agreements)
    return accuracy, agreement_rate

Common pitfalls

• High accuracy artifactually inflates cross-modal agreement because both modalities converge on the correct answer; true alignment requires comparing agreement against a random baseline (p^2 + (1-p)^2/3).
• Assuming vision inputs compensate for language tokenization errors; the paper shows vision can actually degrade performance in domains where text parsing is difficult.
• Ignoring domain-specific tokenization limitations in text modalities (e.g., SMILES, FEN) which independently drive performance gaps unrelated to visual perception.

Evidence (verbatim from paper)

For each sample, the agreement between modalities is binary (either 0 for disagree or 1 for agree), and these binary values are then averaged across all samples to obtain the overall agreement rate. ... GPT-5 demonstrating superior accuracy (0.765) compared to the highest-performing open-source model, Qwen2.5-VL-72B-Instruct (0.514).

Citation

@misc{tang2025seam,
  title={SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models},
  author={Tang et al. (2025)},
  year={2025},
  note={arXiv:2508.18179}
}

• arXiv: 2508.18179