Csvqa Eval

csvqa-eval

CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMs — Ai Jian et al. (arXiv:2505.24120, 2025)

What this evaluates

This benchmark evaluates the scientific reasoning and domain-grounded visual question answering capabilities of Vision-Language Models (VLMs) in Chinese. It probes the ability to integrate multimodal STEM evidence across physics, chemistry, biology, and mathematics with domain knowledge to solve both multiple-choice and open-ended questions.

Datasets

• CSVQA — total 1378; splits: test (1378)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions. Calculated as (number of correct predictions / total number of questions) × 100. Evaluated overall and broken down by subject (Biology, Chemistry, Math, Physics) and question type (Open-ended, Multiple-Choice).

Input / output format

Input: A STEM-related image paired with a Chinese-language question. Questions are either multiple-choice (with one or more correct options, but the model is not told how many) or open-ended.

Output: Multiple-choice: fixed response format parsed deterministically, with a GPT-4o fallback if parsing fails. Open-ended: free-form text response, later scored by GPT-4o.

Scoring recipe

def compute_accuracy(predictions, gold_answers, question_types):
    correct = 0
    for pred, gold, qtype in zip(predictions, gold_answers, question_types):
        if qtype == 'MC':
            parsed = deterministic_parser(pred)
            if parsed is None:
                parsed = gpt4o_fallback_match(pred, gold)
            if parsed == gold:
                correct += 1
        elif qtype == 'Open':
            if gpt4o_judge_consistency(pred, gold):
                correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Models are not informed whether multiple-choice questions have single or multiple correct answers, leading to format confusion and parser failures.
• Open-ended questions are not scored via exact string match; they require an external LLM (GPT-4o) for evaluation, which introduces potential judge bias and variability.
• Deterministic parsing of MC answers may fail on format violations, requiring a GPT-4o fallback mechanism to ensure consistent scoring across all models.

Evidence (verbatim from paper)

For multiple-choice questions, models are constrained to follow a fixed response format using rule-based protocols, with answers extracted via a deterministic parser. Notably, while questions may have single or multiple correct answers, models receive no explicit indication of the number of valid options. If the parser fails due to format violations, we employ a fallback mechanism using GPT-4o to match the answers. Open-ended questions are scored only by GPT-4o to ensure consistency and accuracy of scoring. Table 3 shows that the best-performing model on CSVQA is the closed-source o1, achieving an overall accuracy of 49.6%.

Citation

@misc{ai2025csvqa,
  title={CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMs},
  author={Ai Jian et al.},
  year={2025},
  note={arXiv:2505.24120}
}

• arXiv: 2505.24120