Nuclearqa Eval

nuclearqa-eval

NuclearQA: A Human-Made Benchmark for Language Models for the Nuclear Domain — Acharya et al. (2023) (arXiv:2310.10920, 2023)

What this evaluates

Assesses large language models' deep scientific reasoning and domain-specific knowledge in nuclear physics, chemistry, and material science, without relying on provided context passages. It probes the model's ability to retrieve and apply expert-crafted factual and conceptual knowledge across multiple difficulty levels.

Datasets

• NuclearQA — total 100; splits: test (100); repo https://github.com/pnnl/EXPERT2

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of questions answered correctly. For single-correct and multiple-correct items, all required components must be present. Open-ended/N/A items are excluded from the primary accuracy calculation or scored via expert rubric.

Input / output format

Input: A single question requiring domain knowledge in nuclear-related fields. No supporting text or passages are provided.

Output: Short answers (few words) for 75% of items, or open-ended long answers for 25% of items.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    total = 0
    for pred, gold_ans in zip(predictions, gold):
        if gold_ans['format'] == 'N/A':
            continue
        total += 1
        if is_correct(pred, gold_ans):
            correct += 1
    return correct / total if total > 0 else 0.0

Common pitfalls

• Questions require deep domain knowledge rather than retrieval from provided context passages, so retrieval-augmented models may underperform if not properly grounded.
• Open-ended questions are subjective and marked as N/A in the answer format, making automated exact-match scoring unreliable.
• Multiple-correct answers require all specified components to be present for a full score; partial matches should not be counted as correct.

Evidence (verbatim from paper)

The NuclearQA benchmark presented in this work is a first-of-its-kind benchmark. It has not been adapted from tests originally meant for humans, but is crafted by subject matter experts (SMEs) specifically to assure that these questions are well suited to judge a language model's ability to solve nuclear-related questions. Evaluation reveals that even state-of-the-art LLMs fail to perform adequately, highlighting a significant gap in scientific knowledge representation within current models.

Citation

@misc{acharya2023nuclearqa,
  title={NuclearQA: A Human-Made Benchmark for Language Models for the Nuclear Domain},
  author={Acharya et al. (2023)},
  year={2023},
  note={arXiv:2310.10920}
}

• arXiv: 2310.10920