Scifibench Eval

scifibench-eval

SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation — Roberts et al. (2024) (arXiv:2405.08807, 2024)

What this evaluates

This benchmark evaluates large multimodal models' ability to interpret scientific figures by testing their capacity to match figures to captions and vice versa. It probes fine-grained visual-textual reasoning, attention to scientific details, and robustness against adversarially selected distractors.

Datasets

• SciFIBench — total 2000; splits: test (2000); repo https://github.com/jonathan-roberts1/SciFIBench

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. Calculated as the number of instances where the model's predicted choice letter matches the ground truth, divided by the total number of instances.

Input / output format

Input: For Figure→Caption: a scientific figure image and four multiple-choice caption options. For Caption→Figure: a figure caption and four multiple-choice figure options. Both are provided with a 0-shot chain-of-thought prompt template.

Output: A single choice letter (e.g., 'A', 'B', 'C', or 'D') corresponding to the correct option. Models are instructed to output only the letter, though they often generate verbose reasoning.

Scoring recipe

def compute_accuracy(predictions, gold_letters):
    correct = 0
    for pred, gold in zip(predictions, gold_letters):
        extracted = extract_letter(pred) # via regex or LLM parser
        if extracted == gold:
            correct += 1
    return (correct / len(gold_letters)) * 100

Common pitfalls

• Models frequently ignore the instruction to output only a choice letter, requiring post-hoc parsing (e.g., via an LLM) to extract the answer.
• Performance is highly sensitive to the quality of multiple-choice distractors; adversarial negatives significantly lower accuracy compared to randomly selected ones.
• Shuffling the order of multiple-choice options can cause performance to fluctuate by up to 5%.

Evidence (verbatim from paper)

As expected, in the vast majority of cases, accuracy scores are higher on the random negatives – for some open-source models, the accuracy score more than doubles, and for the closed-source models, the maximum accuracy score is almost met.

Citation

@misc{roberts2024scifibench,
  title={SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation},
  author={Roberts et al. (2024)},
  year={2024},
  note={arXiv:2405.08807}
}

• arXiv: 2405.08807