Sciqag 24d Eval

sciqag-24d-eval

SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluation — Wan et al. (2024) (arXiv:2405.09939, 2024)

What this evaluates

Evaluates open-ended, closed-book scientific question answering capabilities. It probes a model's ability to generate comprehensive, accurate, and reasonable answers to research-level science questions without external context or reference papers.

Datasets

• SciQAG-24D — total 188042; splits: train (-1), test (-1)
• SciQ — total 11000; splits: train (10000), test (1000)

Metrics

• CAR (primary) — range: [0, 5]
  • A modified LLM-judged metric derived from RACAR, explicitly excluding Relevance and Agnosticism. It scores generated answers on three dimensions: Completeness, Accuracy, and Reasonableness, typically on a 5-point scale.
• accuracy — range: [0, 1]
  • Standard exact-match accuracy for multiple-choice questions.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall for classification tasks.
• MAE — range: other
  • Mean Absolute Error for regression tasks.
• KL divergence — range: [0, inf)
  • Kullback-Leibler divergence for transformation tasks.

Input / output format

Input: Open-ended science questions generated from scientific literature, provided without context or reference papers.

Output: Open-ended textual answers to the science questions.

Scoring recipe

def score_car(question, answer):
    prompt = f'Question: {question}\nAnswer: {answer}\nScore Completeness, Accuracy, Reasonableness on 1-5 scale.'
    llm_response = call_llm(prompt)
    scores = extract_scores(llm_response)
    return sum(scores) / len(scores)

def score_accuracy(prediction, gold):
    return 1.0 if prediction.strip().lower() == gold.strip().lower() else 0.0

def score_f1(prediction, gold):
    pred_set = set(prediction.lower().split())
    gold_set = set(gold.lower().split())
    if not pred_set and not gold_set: return 1.0
    prec = len(pred_set & gold_set) / len(pred_set) if pred_set else 0
    rec = len(pred_set & gold_set) / len(gold_set) if gold_set else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• CAR is a modified version of RACAR that explicitly excludes 'Relevance' and 'Agnosticism'; using the full RACAR framework will yield incorrect scores.
• The CAR metric relies on LLM-as-a-judge evaluation, so results are highly sensitive to the judge model, prompt phrasing, and temperature settings.
• SciQAG-24D questions are auto-generated and open-ended; exact string matching is inappropriate, requiring semantic or LLM-based scoring.

Evidence (verbatim from paper)

For the evaluation of the quality of generated answers in the zero-shot experiment, we adopt a modified version of the RACAR framework. We exclude the criteria of "Relevance" and "Agnosticism", which are specifically designed to assess the alignment of question-answer (QA) pairs with a given paper, and name this modified metric as CAR (see [Appendix J]).

Citation

@misc{wan2024sciqag,
  title={SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluation},
  author={Wan et al. (2024)},
  year={2024},
  note={arXiv:2405.09939}
}

• arXiv: 2405.09939