Scienceqa Eval

scienceqa-eval

Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering — Lu et al. (2022) (arXiv:2209.09513, 2022)

What this evaluates

Evaluates multimodal reasoning and scientific question answering by requiring models to process questions, images, and context to select correct multiple-choice answers. It also probes chain-of-thought reasoning capabilities by measuring the quality of generated explanations and lectures.

Datasets

• ScienceQA — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted options out of total test instances. For text generation models, the most similar option is selected as the final prediction to compute accuracy.
• BLEU-1 — range: [0, 1]
  • Standard unigram precision metric for evaluating generated text against gold explanations.
• BLEU-4 — range: [0, 1]
  • Standard four-gram precision metric for evaluating generated text against gold explanations.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence recall/precision metric for evaluating generated text against gold explanations.
• Similarity — range: [0, 1]
  • Cosine similarity of semantic embeddings between generated and gold explanations, computed using the Sentence-BERT network.
• Human Relevance/Correctness/Completeness — range: percent
  • Annotator ratings assessing whether generated explanations are relevant, correct, and complete.

Input / output format

Input: Question text, multiple-choice options, contextual information (text or image), and optionally lecture/explanation text in few-shot prompts.

Output: For classification baselines: a single selected option letter/label. For generation baselines: a text string containing the answer, optionally followed by a lecture and/or explanation.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return (correct / len(golds)) * 100

def get_final_prediction(generated_text, options):
    return max(options, key=lambda opt: similarity(generated_text, opt))

bleu1 = compute_bleu(generated, gold, n=1)
rouge_l = compute_rouge(generated, gold, rouge_type='L')
sim = cosine_similarity(sentence_bert(generated), sentence_bert(gold))

Common pitfalls

• Generation models must explicitly map their free-text output to the closest multiple-choice option to compute accuracy; raw text generation is not directly scored.
• The position of generated lectures/explanations relative to the answer critically affects performance; generating long reasoning text before the answer can cause early stopping or token limit exhaustion.
• Automatic metrics (BLEU, ROUGE, Similarity) only capture partial quality of explanations and must be supplemented with human evaluation for relevance, correctness, and completeness.

Evidence (verbatim from paper)

The heuristics and VQA baselines treat our ScienceQA task as a multi-class classification problem with multiple options and are evaluated with the accuracy metrics. UnifiedQA and GPT-3 treat ScienceQA as a text generation problem. So the most similar option is selected as the final prediction to evaluate the question answering accuracy. We further report automatic metrics (BLEU-1/4, ROUGE-L, and (sentence) Similarity to evaluate the generated lectures and explanations... The Similarity metric computes the cosine-similarity of semantic embeddings between two sentences based on the Sentence-BERT network.

Citation

@misc{lu2022learn,
  title={Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering},
  author={Lu et al. (2022)},
  year={2022},
  note={arXiv:2209.09513}
}

• arXiv: 2209.09513