Sc Heureka Bench Eval

sc-heureka-bench-eval

HeurekaBench: A Benchmarking Framework for AI Co-scientist — Panigrahi et al. (2026) (arXiv:2601.01678, 2026)

What this evaluates

Evaluates AI co-scientist agents' ability to autonomously plan, execute, and interpret single-cell biology workflows to answer open-ended research questions (OEQs) and multiple-choice questions (MCQs). It probes hypothesis generation, code execution, data analysis, and scientific reasoning in a domain-specific setting.

Datasets

• sc-HeurekaBench-Lite — total 40; splits: test (40); repo https://github.com/mlbio-epfl/heurekabench

Metrics

• Correctness [1-5] (primary) — range: [1, 5]
  • Ordinal score from 1 to 5 assigned by an LLM judge (GPT-4o) or human experts, evaluating how well the agent's response and workflow address the research question and ground truth.
• Accuracy [%] — range: [0, 100]
  • Percentage of correctly answered multiple-choice questions (MCQs).
• Recall [%] — range: [0, 100]
  • Percentage of relevant items or correct components retrieved/identified by the agent out of all relevant items in the ground truth.
• Precision [%] — range: [0, 100]
  • Percentage of items identified by the agent that are actually relevant or correct according to the ground truth.

Input / output format

Input: Experimental single-cell data (datasets < 750 MB) paired with an open-ended research question (OEQ) or multiple-choice question (MCQ).

Output: Autonomous multi-step workflow including code execution, data analysis, and a final textual answer or hypothesis addressing the research question.

Scoring recipe

if question_type == 'OEQ':
    score = llm_judge.evaluate(response, ground_truth, scale=[1,5])
    return score
else:
    predictions = extract_mcq_answer(response)
    gold = get_mcq_ground_truth(question)
    acc = (predictions == gold).mean() * 100
    recall = tp / (tp + fn) * 100
    precision = tp / (tp + fp) * 100
    return acc, recall, precision

Common pitfalls

• Agents may crash or exceed time/API limits on large datasets; the benchmark restricts evaluation to the 'Lite' subset (<750 MB) for fair comparison.
• Correctness scores are ordinal and subject to LLM-judge bias; the paper validates alignment with human experts (Spearman ρ=0.90-0.93) but notes stochasticity from critic modules.
• Task prompts and agent configurations (e.g., step limits, critic placement) significantly impact results and must be standardized across runs.

Evidence (verbatim from paper)

We benchmark three state-of-the-art agents for single-cell biology on the sc-HeurekaBench: Biomni*(huang2025biomni), CellVoyager(alber2025cellvoyager), and BixBench-Agent(mitchener2025bixbench)*. ... We report results on questions related to datasets smaller than 750 MB, which we term sc-HeurekaBench-Lite. This subset contains 22 out of 50 OEQs and 18 out of 50 MCQs on which all agents could run. ... Correctness [1-5] | Accuracy [%] | Recall [%] | Precision [%]

Citation

@misc{panigrahi2026heurekabench,
  title={HeurekaBench: A Benchmarking Framework for AI Co-scientist},
  author={Panigrahi et al. (2026)},
  year={2026},
  note={arXiv:2601.01678}
}

• arXiv: 2601.01678