Hiscibench Eval

hiscibench-eval

HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discovery — Zhang et al. (2025) (arXiv:2512.22899, 2025)

What this evaluates

Evaluates large language models across five hierarchical cognitive stages of scientific inquiry, ranging from foundational factual recall and literature parsing to advanced synthesis, literature review generation, and data-driven scientific discovery. It probes multimodal comprehension, cross-lingual reasoning, and computational problem-solving across six scientific disciplines.

Datasets

• HiSciBench — total 8735; splits: test (8735)

Metrics

• accuracy (primary) — range: percent
  • Proportion of correctly answered instances out of the total. For multiple-choice and QA tasks (L1–L3), it is exact-match accuracy. For generation tasks (L4–L5), correctness is determined by matching against expert-annotated reference solutions or automated execution/judgment.

Input / output format

Input: Varies by level: L1 provides multiple-choice questions; L2 provides multimodal scientific documents (images of pages, LaTeX/Markdown); L3 provides multimodal literature with questions; L4 provides research topic keywords and core papers; L5 provides problem descriptions, data files, and domain knowledge.

Output: L1–L3: selected option or generated answer; L4: comprehensive literature review text; L5: Python code, execution results, and interpretation.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if pred == gold_item:  # exact match for QA
            correct += 1
        elif is_correct_generation(pred, gold_item):  # L4/L5 expert/LLM judgment
            correct += 1
    return (correct / len(gold)) * 100

Common pitfalls

• Models must handle complex multimodal layouts (equations, figures) in L2, not just plain text.
• L5 requires executable Python code and data analysis, not just conceptual answers.
• Cross-lingual tasks (L2.2, L3.2) require preserving technical semantics and symbolic accuracy across languages.

Evidence (verbatim from paper)

state-of-the-art models achieve 69% accuracy on foundational tasks but drop to 25% on discovery-level reasoning, highlighting a critical gap in synthesis and creativity.

Citation

@misc{zhang2025hiscibench,
  title={HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discovery},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2512.22899}
}

• arXiv: 2512.22899