Miss QA Eval

miss-qa-eval

Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers — Zhao et al. (2025) (arXiv:2507.10787, 2025)

What this evaluates

Evaluates multimodal foundation models' ability to interpret schematic diagrams in scientific papers and answer information-seeking questions based on visual-textual context. It also probes models' robustness in identifying unanswerable questions when sufficient information is absent.

Datasets

• MISS-QA — total 1500; splits: testmini (500), test (1000)

Metrics

• accuracy (primary) — range: [0, 1]
  • Computed via an LLM-as-Judge framework (GPT-4.1) that compares the model's generated response to the ground-truth answer, assigning a score of 0, 0.5, or 1 per instance. The final metric is the average score across all instances.

Input / output format

Input: A schematic diagram image, a natural language question, and the relevant textual context from the scientific paper.

Output: A natural language text response.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    scores = []
    for pred, gold in zip(predictions, gold_answers):
        # GPT-4.1 LLM-as-Judge compares pred and gold
        score = llm_judge(prompt=f'Compare response: {pred} with ground truth: {gold}')
        # Returns 0, 0.5, or 1
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• Models frequently exhibit overconfidence on unanswerable questions, generating responses instead of correctly indicating that the information is missing.
• Models often over-rely on isolated visual interpretation of diagrams while neglecting the provided paper context, or fail to interpret diagram structures correctly.

Evidence (verbatim from paper)

We use accuracy as the primary metric to evaluate model performance on MISS-QA. Following recent benchmark studies (Lu et al., 2024; Li et al., 2024a), we adopt the LLM-as-Judge framework with GPT-4.1 as the base evaluator. Specifically, the model is prompted to assign an accuracy score of 0, 0.5, or 1 by comparing the generated response with the ground-truth answer.

Citation

@misc{zhao2025missqa,
  title={Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2507.10787}
}

• arXiv: 2507.10787