Sciknoweval Eval

sciknoweval-eval

SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models — Feng et al. (2024) (arXiv:2406.09098, 2024)

What this evaluates

Probes large language models' scientific knowledge across five progressive cognitive levels: memory, comprehension, reasoning, ethical discernment, and real-world application. Covers four scientific domains (biology, chemistry, physics, materials science) using diverse question formats including multiple-choice, relation extraction, and open-ended protocol design.

Datasets

• SciKnowEval — total 28000; splits: test (28000); repo https://github.com/OpenDFM/SciEval

Metrics

• accuracy — range: [0, 1]
  • Proportion of correctly predicted answers for True/False, classification, and multiple-choice questions.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall for relation extraction tasks.
• GPT-4o rating (normalized) — range: [0, 1]
  • LLM-as-a-judge scores generative responses on a 1–5 scale, then linearly mapped to [0, 1].
• overall normalized score (primary) — range: [0, 1]
  • Average of all task-level scores after each individual metric (accuracy, F1, GPT-4o rating) has been normalized to [0, 1].

Input / output format

Input: A system prompt outlining question types and categories, followed by a single question instance in a zero-shot setting (no few-shot examples).

Output: Model-generated text response, which may be a direct answer (for MC/TF), extracted relations, or a detailed generative output (e.g., experimental protocol).

Scoring recipe

def compute_overall_score(predictions, golds, task_types):
    normalized_scores = []
    for pred, gold, t in zip(predictions, golds, task_types):
        if t in ['mc', 'tf', 'cls']:
            score = 1.0 if pred == gold else 0.0
        elif t == 'rel':
            score = f1_score(pred, gold)
        elif t == 'gen':
            score = (gpt4o_judge(pred) - 1) / 4  # maps 1-5 to 0-1
        normalized_scores.append(score)
    return sum(normalized_scores) / len(normalized_scores)

Common pitfalls

• Failing to strictly enforce zero-shot evaluation, which inflates performance by providing in-context examples.
• Averaging raw metric values (e.g., mixing accuracy with F1 or LLM scores) without first normalizing each to [0, 1], leading to biased overall scores.
• Using exact-match or keyword-based grading for generative tasks instead of the specified GPT-4o LLM-judge rubric.

Evidence (verbatim from paper)

When evaluating True/False, classification and multiple-choice questions, we use accuracy as the performance metric. For relation extraction questions, we use the $F_{1}$-score that combines precision and recall. For generative questions, we designed meticulous prompts for GPT-4o to evaluate the responses of LLMs. The scoring prompt templates are exhibited in [A9]. We normalize the results of all evaluation metrics to the range of 0 to 1. We then compute the average score for each level, as well as the overall average score across all levels.

Citation

@misc{feng2024sciknoweval,
  title={SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models},
  author={Feng et al. (2024)},
  year={2024},
  note={arXiv:2406.09098}
}

• arXiv: 2406.09098