# Scienceqa Eval

> Evaluates multimodal reasoning and scientific question answering by requiring models to process questions, images, and context to select correct multiple-choice answers. It also probes chain-of-thought reasoning capabilities by measuring the quality of generated explanations and lectures. Use when the user wants to benchmark on ScienceQA, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/scienceqa-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/scienceqa-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/scienceqa-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/scienceqa-eval

---


# scienceqa-eval

> Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering — Lu et al. (2022) (arXiv:2209.09513, 2022)

## What this evaluates

Evaluates multimodal reasoning and scientific question answering by requiring models to process questions, images, and context to select correct multiple-choice answers. It also probes chain-of-thought reasoning capabilities by measuring the quality of generated explanations and lectures.

## Datasets

- **ScienceQA** — total ?; splits: train (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted options out of total test instances. For text generation models, the most similar option is selected as the final prediction to compute accuracy.
- `BLEU-1` — range: [0, 1]
  - Standard unigram precision metric for evaluating generated text against gold explanations.
- `BLEU-4` — range: [0, 1]
  - Standard four-gram precision metric for evaluating generated text against gold explanations.
- `ROUGE-L` — range: [0, 1]
  - Longest common subsequence recall/precision metric for evaluating generated text against gold explanations.
- `Similarity` — range: [0, 1]
  - Cosine similarity of semantic embeddings between generated and gold explanations, computed using the Sentence-BERT network.
- `Human Relevance/Correctness/Completeness` — range: percent
  - Annotator ratings assessing whether generated explanations are relevant, correct, and complete.

## Input / output format

**Input**: Question text, multiple-choice options, contextual information (text or image), and optionally lecture/explanation text in few-shot prompts.

**Output**: For classification baselines: a single selected option letter/label. For generation baselines: a text string containing the answer, optionally followed by a lecture and/or explanation.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return (correct / len(golds)) * 100

def get_final_prediction(generated_text, options):
    return max(options, key=lambda opt: similarity(generated_text, opt))

bleu1 = compute_bleu(generated, gold, n=1)
rouge_l = compute_rouge(generated, gold, rouge_type='L')
sim = cosine_similarity(sentence_bert(generated), sentence_bert(gold))
```

## Common pitfalls

- Generation models must explicitly map their free-text output to the closest multiple-choice option to compute accuracy; raw text generation is not directly scored.
- The position of generated lectures/explanations relative to the answer critically affects performance; generating long reasoning text before the answer can cause early stopping or token limit exhaustion.
- Automatic metrics (BLEU, ROUGE, Similarity) only capture partial quality of explanations and must be supplemented with human evaluation for relevance, correctness, and completeness.

## Evidence (verbatim from paper)

> The heuristics and VQA baselines treat our ScienceQA task as a multi-class classification problem with multiple options and are evaluated with the accuracy metrics. UnifiedQA and GPT-3 treat ScienceQA as a text generation problem. So the most similar option is selected as the final prediction to evaluate the question answering accuracy. We further report automatic metrics (BLEU-1/4, ROUGE-L, and (sentence) Similarity to evaluate the generated lectures and explanations... The Similarity metric computes the cosine-similarity of semantic embeddings between two sentences based on the Sentence-BERT network.

## Citation

```bibtex
@misc{lu2022learn,
  title={Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering},
  author={Lu et al. (2022)},
  year={2022},
  note={arXiv:2209.09513}
}
```

- arXiv: 2209.09513