Spiqa Eval

spiqa-eval

SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers — Pramanick et al. (2024) (arXiv:2407.09413, 2024)

What this evaluates

Evaluates multimodal long-context reasoning and figure/table comprehension on scientific papers. Tests direct question answering with images, full paper context, and chain-of-thought retrieval capabilities.

Datasets

• SPIQA — total ?; splits: train (-1), test-A (-1), test-B (-1), test-C (-1); repo https://github.com/google/spiqa

Metrics

• L3Score (primary) — range: other
  • A log-likelihood-based metric that computes token probabilities from an LLM to assess answer confidence and semantic equivalence, outperforming traditional token-matching metrics.
• METEOR — range: [0, 1]
  • Standard machine translation metric measuring alignment between generated and reference answers based on synonyms, stems, and exact matches.
• CIDEr — range: [0, 1]
  • Consensus-based Image Description Evaluation metric that weights n-grams by their IDF scores to measure consensus with reference answers.
• ROUGE-L — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation measuring the longest common subsequence between generated and reference answers.
• BERTScore F1 — range: [0, 1]
  • F1 score computed from contextual embeddings of generated and reference answers using a pre-trained BERT model.
• Top-1 Retrieval Accuracy — range: [0, 1]
  • Percentage of questions where the model correctly retrieves the single most helpful reference image.

Input / output format

Input: Question text, reference images (figures/tables), and optionally full paper text. For open-weight models, images are provided one-by-one in a multi-turn setup.

Output: Free-form natural language answer. For CoT QA, step-by-step reasoning followed by the final answer, plus retrieval of reference images.

Scoring recipe

def evaluate(predictions, golds, questions, images, metric):
    scores = []
    for pred, gold, q, img in zip(predictions, golds, questions, images):
        if metric == 'L3Score':
            ll = compute_log_likelihood(pred, context=(q, img))
            scores.append(aggregate_ll(ll))
        elif metric == 'Top-1 Retrieval Accuracy':
            scores.append(1.0 if pred == gold else 0.0)
        else:
            scores.append(compute_standard_metric(pred, gold, metric))
    return mean(scores)

Common pitfalls

• Omitting figure/table captions significantly drops performance for all models.
• Traditional metrics like ROUGE-L and BERTScore fail to correctly score semantically correct but lexically different answers.
• Models struggle with complex plots, charts, and tables requiring mathematical reasoning.

Evidence (verbatim from paper)

For evaluating the free-form answers, we report five different metrics for comprehensive analysis - METEOR, CIDEr, ROUGE-L, BERTScore F1 and the proposed L3Score. For the CoT QA task, we also report the top-1 accuracy for retrieving the helpful images to answer the question.

Citation

@misc{pramanick2024spiqa,
  title={SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers},
  author={Pramanick et al. (2024)},
  year={2024},
  note={arXiv:2407.09413}
}

• arXiv: 2407.09413