# Scp 116k Eval

> Evaluates the scientific reasoning and problem-solving capabilities of LLMs on graduate-level higher education science problems. It measures how well models can parse and solve complex scientific questions involving formulas and equations. Use when the user wants to benchmark on SCP-116K, or asks about evaluating this task. Reports Accuracy.

- Skill: `qhjqhj00/scp-116k-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/scp-116k-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/scp-116k-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/scp-116k-eval

---


# scp-116k-eval

> SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domain — Lu et al. (2025) (arXiv:2501.15587, 2025)

## What this evaluates

Evaluates the scientific reasoning and problem-solving capabilities of LLMs on graduate-level higher education science problems. It measures how well models can parse and solve complex scientific questions involving formulas and equations.

## Datasets

- **SCP-116K** — total 116756; splits: (unstated); repo https://github.com/AQA6666/SCP-116K-open

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly solved problems out of the total number of problems in the dataset.

## Input / output format

**Input**: Scientific problem statements, often containing complex formulas and equations, extracted from higher education science materials.

**Output**: Not explicitly specified in the provided text.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold):
    correct = sum(1 for pred, gold in zip(predictions, gold) if pred == gold)
    return (correct / len(gold)) * 100
```

## Common pitfalls

- The evaluation section does not specify train/val/test splits for SCP-116K, making reproducibility of baseline splits difficult.
- Distillation experiments evaluate final performance on GPQA-diamond rather than SCP-116K, which may not directly reflect improvement on the target dataset.

## Evidence (verbatim from paper)

> These models achieve accuracy rates of 58.40% and 55.79%, respectively, on SCP-116K.

## Citation

```bibtex
@misc{lu2025scp116k,
  title={SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domain},
  author={Lu et al. (2025)},
  year={2025},
  note={arXiv:2501.15587}
}
```

- arXiv: 2501.15587