Scievalkit Eval

scievalkit-eval

SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence — Wang et al. (2025) (arXiv:2512.22334, 2025)

What this evaluates

Evaluates large language models' scientific intelligence across seven core dimensions, including multimodal perception, understanding, reasoning, knowledge comprehension, code generation, symbolic reasoning, and hypothesis generation. It covers multiple scientific disciplines using both text-only and multimodal inputs to assess real-world scientific workflow capabilities.

Datasets

• SLAKE — total ?; splits: test (-1)
• MSEarth — total ?; splits: test (-1)
• SFE — total ?; splits: test (-1)
• OmniEarth — total ?; splits: test (-1)
• OmniMedVQA — total ?; splits: test (-1)
• PhyX — total ?; splits: test (-1)
• ChemBench — total ?; splits: test (-1)
• ChemBench4K — total ?; splits: test (-1)
• LLM4Chem — total ?; splits: test (-1)
• ClimaQA — total ?; splits: test (-1)
• EarthSE — total ?; splits: test (-1)
• ProteinLMBench — total ?; splits: test (-1)
• BioProbench — total ?; splits: test (-1)
• MaScQA — total ?; splits: test (-1)
• TRQA — total ?; splits: test (-1)
• Biology-Instructions — total ?; splits: test (-1)
• Mol-Instructions — total ?; splits: test (-1)
• PEER — total ?; splits: test (-1)
• SciCode — total ?; splits: test (-1)
• AstroVisBench — total ?; splits: test (-1)
• CMPhysBench — total ?; splits: test (-1)
• PHYSICS — total ?; splits: test (-1)
• ResearchBench — total ?; splits: test (-1)

Metrics

• scoring criteria (primary) — range: percent
  • Task-specific evaluation metrics applied per benchmark type, including exact-match for multiple-choice, execution pass rate for code, and expert-graded scoring for open-ended or free-form generation tasks.

Input / output format

Input: Text-only: domain-specific scientific questions or prompts. Multimodal: images (e.g., diagrams, plots, radiological imagery, molecular structures) paired with textual context or questions.

Output: Model-generated responses, which may be multiple-choice selections, free-form scientific explanations, executable code snippets, or structured reasoning steps.

Scoring recipe

def evaluate(predictions, gold, task_type):
    if task_type == 'multiple_choice':
        return 1.0 if predictions == gold else 0.0
    elif task_type == 'code':
        return execute_and_check(predictions, gold)
    elif task_type == 'open_ended':
        return expert_or_llm_grader(predictions, gold)
    return 0.0

Common pitfalls

• Assuming a single uniform metric applies across all modalities and task types, ignoring the suite's task-specific scoring criteria.
• Overlooking the requirement for expert calibration and scientific validity, which leads to superficial evaluation of complex reasoning tasks.
• Treating multimodal inputs as purely textual, missing critical visual scientific information required for accurate scoring.

Evidence (verbatim from paper)

Each benchmark undergoes manual verification and calibration by domain experts, who validate the correctness of task formulations, solution rationales, and scoring criteria. The suite covers multiple modalities (text, diagrams, molecular structures, protein sequences, scientific plots, radiological imagery, etc.) and multiple task formats (multiple-choice, free-form generation, code execution, document analysis), thereby capturing the multimodal and procedural nature of real scientific workflows.

Citation

@misc{wang2025scievalkit,
  title={SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2512.22334}
}

• arXiv: 2512.22334