Scigraphqa Eval

scigraphqa-eval

SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs — Li et al. (2023) (arXiv:2308.03349, 2023)

What this evaluates

Evaluates multi-modal large language models' ability to interpret scientific graphs and generate accurate, context-aware answers in a multi-turn conversational setting. It probes open-vocabulary visual reasoning and the model's capacity to leverage auxiliary paper metadata for grounded responses.

Datasets

• SciGraphQA — total 295000; splits: test (3000)

Metrics

• CIDEr (primary) — range: [0, 1]
  • Computes TF-IDF weighted n-gram similarity between predicted and reference answers, emphasizing rare but informative words to better capture consensus with human references.

Input / output format

Input: Scientific graph image, contextual paper metadata (title, abstract, paragraph, graph text), and a multi-turn question.

Output: Textual answer string.

Scoring recipe

def compute_cider(predictions, references):
    scores = []
    for pred, ref in zip(predictions, references):
        pred_tokens = tokenize(pred)
        ref_tokens = tokenize(ref)
        score = cider_score(pred_tokens, ref_tokens)
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• Models often fail to maintain multi-turn context, answering only the latest question while ignoring prior turns.
• Injecting large amounts of paper metadata into prompts can overwhelm the model's attention mechanism, degrading graph-focused reasoning.
• BLEU and ROUGE may unfairly penalize technically correct but paraphrased scientific answers due to strict n-gram matching.

Evidence (verbatim from paper)

We set aside a test set of 3K samples and computed NLP metrics including BLEU-4, ROUGE, and CIDEr.

Citation

@misc{li2023scigraphqa,
  title={SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2308.03349}
}

• arXiv: 2308.03349