Scivqa Eval

scivqa-eval

NFDI4DS Shared Tasks for Scholarly Document Processing — Abu Ahmad et al. (2025) (arXiv:2509.22141, 2025)

What this evaluates

Evaluates multimodal LLMs on closed-ended visual and non-visual question answering over scientific figures. It probes recognition of visual attributes (color, shape, position) and reasoning capabilities across diverse chart types.

Datasets

• SciVQA — total 21000; splits: train (-1), test (-1); HF katebor/SciVQA; repo https://sdproc.org/2025/scivqa.html

Metrics

• ROUGE-1 F1 (primary) — range: [0, 1]
  • Unigram overlap F1 score between the predicted answer and the reference answer.

Input / output format

Input: Image of a scientific figure, its caption, metadata, and a closed-ended question.

Output: Textual answer to the question.

Scoring recipe

pred_answer = model.generate(image, caption, metadata, question)
gold_answer = gold_label
rouge1_f1 = rouge1_score(gold_answer, pred_answer)
rougeL_f1 = rougeL_score(gold_answer, pred_answer)
bertscore_f1 = bertscore_f1(gold_answer, pred_answer)

Common pitfalls

• The dataset mixes visual and non-visual questions, but evaluation is often reported as an aggregate score without separating them.
• Metrics include ROUGE-1, ROUGE-L, and BertScore, which measure different aspects of text similarity and may not align perfectly with factual correctness.

Evidence (verbatim from paper)

The evaluation is based on precision, recall and F1-scores of ROUGE-1, ROUGE-L, and BertScore. The highest-ranking team achieved scores of 0.80 for ROUGE-1 and ROUGE-L and 0.98 for BertScore.

Citation

@misc{abuhmad2025nfdi4ds,
  title={NFDI4DS Shared Tasks for Scholarly Document Processing},
  author={Abu Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2509.22141}
}

• arXiv: 2509.22141