Sciex Eval

sciex-eval

SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading — Dinh et al. (2024) (arXiv:2406.10421, 2024)

What this evaluates

Evaluates large language models on solving university-level scientific exams in computer science. It probes capabilities in open-ended reasoning, mathematical proof writing, long-form explanations, and multimodal (image-text) understanding across English and German languages.

Datasets

• SciEx — total ?; splits: test (-1); repo https://github.com/TuAnh23/SciEx

Metrics

• Normalized score (0-100%) (primary) — range: percent
  • Calculated as (raw expert-assigned score / maximum possible score) * 100. Maximum scores are predefined by lecturers for each exam or question.
• Pearson correlation — range: [-1, 1]
  • Measures the linear correlation between LLM-as-a-judge scores and expert grading scores on normalized or original scales.

Input / output format

Input: Open-ended scientific exam questions, which may be text-only or multimodal (text + images), in English or German. Models receive the question prompt via API or local inference.

Output: Freeform text responses (mathematical proofs, explanations, code, or descriptions). For multimodal questions, models may generate text descriptions of drawings or figure paths.

Scoring recipe

def compute_metric(predictions, gold):
    raw_score = gold['expert_raw_score']
    max_score = gold['max_possible_score']
    normalized = (raw_score / max_score) * 100
    return normalized

Common pitfalls

• LLMs often ignore brevity constraints and output lengthy answers, which can negatively impact content quality scores compared to concise student answers.
• Human-labeled difficulty levels do not always align with LLM performance; stronger models may outperform students on 'hard' questions while weaker models struggle disproportionately.
• Text-only LLMs are excluded from image-related questions, but even multimodal models show a larger performance gap on image questions compared to text-only ones.

Evidence (verbatim from paper)

We normalize the grade to be between 0 and 100%, since they have different scales. The normalization is done by taking the scores obtained by the examinee divided by the maximum score possible per exam/question, where the maximum scores possible are predefined by the lecturers. We use Pearson correlation on the normalized scores as our metric.

Citation

@misc{dinh2024sciex,
  title={SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading},
  author={Dinh et al. (2024)},
  year={2024},
  note={arXiv:2406.10421}
}

• arXiv: 2406.10421