qhjqhj00 scp-116k-eval
SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domain — Lu et al. (2025) (arXiv:2501.15587, 2025)
What this evaluates
Evaluates the scientific reasoning and problem-solving capabilities of LLMs on graduate-level higher education science problems. It measures how well models can parse and solve complex scientific questions involving formulas and equations.
Datasets
- SCP-116K — total 116756; splits: (unstated); repo https://github.com/AQA6666/SCP-116K-open
Metrics
Accuracy(primary) — range: percent- Percentage of correctly solved problems out of the total number of problems in the dataset.
Input / output format
Input: Scientific problem statements, often containing complex formulas and equations, extracted from higher education science materials.
Output: Not explicitly specified in the provided text.
Scoring recipe
def calculate_accuracy(predictions, gold):
correct = sum(1 for pred, gold in zip(predictions, gold) if pred == gold)
return (correct / len(gold)) * 100
Common pitfalls
- The evaluation section does not specify train/val/test splits for SCP-116K, making reproducibility of baseline splits difficult.
- Distillation experiments evaluate final performance on GPQA-diamond rather than SCP-116K, which may not directly reflect improvement on the target dataset.
Evidence (verbatim from paper)
These models achieve accuracy rates of 58.40% and 55.79%, respectively, on SCP-116K.
Citation
@misc{lu2025scp116k,
title={SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domain},
author={Lu et al. (2025)},
year={2025},
note={arXiv:2501.15587}
}
- arXiv: 2501.15587