Bioagent Bench Eval

bioagent-bench-eval

BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics — Fa et al. (2026) (arXiv:2601.21800, 2026)

What this evaluates

Evaluates AI agents' ability to execute multi-step bioinformatics pipelines (e.g., RNA-seq, variant calling) under normal and perturbed conditions. It probes step-level reasoning, tool-use robustness, and the capacity to produce correctly formatted final artifacts despite input corruption or prompt bloat.

Datasets

• BioAgent Bench Tasks — total ?; splits: test (-1)

Metrics

• completion rate (%) (primary) — range: percent
  • Percentage of required pipeline steps that pass the LLM grader's check for artifact generation and format compliance.
• f1_score — range: [0, 1]
  • F1-score computed where applicable (only for the GIAB dataset).
• final_result_reached — range: boolean
  • Binary flag indicating whether the agent produced the final requested result artifact.
• results_match — range: boolean
  • Task-specific correctness flag derived from rubric rules.

Input / output format

Input: System prompt, input data files, and prompt instructions specifying the goal and expected output format (CSV/TSV).

Output: Generated intermediate output files, a final result file (CSV/TSV table as text), and an execution trace (folders and file paths).

Scoring recipe

def grade_run(input_files, reference_data, expected_outcome, agent_outcome, agent_trace, grading_prompt):
    # LLM Grader (GPT-5.1) evaluates against rubric
    steps_completed = llm_output.steps_completed
    final_result_reached = llm_output.final_result_reached
    results_match = llm_output.results_match
    f1_score = llm_output.f1_score if dataset == 'giab' else None
    return steps_completed, final_result_reached, results_match, f1_score

def completion_rate(task_steps, grading_results):
    passed_steps = sum(1 for step in task_steps 
                       if grading_results[step].final_result_reached 
                       and grading_results[step].results_match)
    return (passed_steps / len(task_steps)) * 100

Common pitfalls

• Multiple valid solution paths and tool choices mean there is no single canonical ground truth, making deterministic scoring impossible.
• Performance is tightly coupled to the specific harness (e.g., Claude Code, Codex CLI) and sandbox configuration, not just the base LLM.
• LLM grading introduces subjectivity and potential bias, especially when evaluating numerical accuracy versus pipeline completion.

Evidence (verbatim from paper)

Our primary metric is completion rate (%). For each task, we evaluate whether the agent completes each required pipeline step and produces the requested final artifact in the specified format (CSV/TSV). The completion rate is the percentage of required steps that pass this check, as assessed by the LLM grader.

Citation

@misc{fa2026bioagentbench,
  title={BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics},
  author={Fa et al. (2026)},
  year={2026},
  note={arXiv:2601.21800}
}

• arXiv: 2601.21800