Lab Bench Eval

lab-bench-eval

LAB-Bench: Measuring Capabilities of Language Models for Biology Research — Laurent et al. (2024) (arXiv:2407.10362, 2024)

What this evaluates

This benchmark evaluates large language models' capabilities in biology research, including literature retrieval, figure/table interpretation, database querying, protocol troubleshooting, and DNA/protein sequence manipulation. It probes whether models can perform multi-step, tool-dependent scientific reasoning or rely on memorization and heuristic guesswork.

Datasets

• LAB-Bench — total 2457; splits: test (2457)

Metrics

• accuracy (primary) — range: [0, 1]
  • Number of questions answered correctly divided by the total number of questions (correct / all).
• precision — range: [0, 1]
  • Number of questions answered correctly divided by the number of questions attempted or covered (correct / attempted).

Input / output format

Input: Multiple-choice questions tailored to specific subtasks. Inputs vary by category: plain text for LitQA2/SuppQA/ProtocolQA, images of figures/tables for FigQA/TableQA, biological sequences and enzyme details for SeqQA/CloningScenarios, and database identifiers for DbQA. Models receive a question, multiple options, and an explicit option to decline answering.

Output: Free-text completion containing the selected multiple-choice option letter/text, or a decline statement. Answers are parsed directly from the text output.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    total = len(gold_labels)
    attempted = sum(1 for p in predictions if p is not None and p != 'decline')
    accuracy = correct / total if total > 0 else 0.0
    precision = correct / attempted if attempted > 0 else 0.0
    return {'accuracy': accuracy, 'precision': precision}

Common pitfalls

• Models often achieve high accuracy through distractor elimination heuristics or memorization rather than genuine reasoning or tool use.
• High coverage rates can mask poor reasoning; models may guess frequently, inflating accuracy while lacking reliability.
• Multi-modal subtasks (FigQA, TableQA) require image inputs, but text-only models cannot be evaluated on them, leading to missing data if not handled.

Evidence (verbatim from paper)

Therefore, for each question, models were given a specific option to decline to answer for lack of information. This allows us to report performance both as accuracy (number of questions correct / number of questions) and precision (number of questions correct / coverage) where coverage is (number of questions attempted).

Citation

@misc{laurent2024labbench,
  title={LAB-Bench: Measuring Capabilities of Language Models for Biology Research},
  author={Laurent et al. (2024)},
  year={2024},
  note={arXiv:2407.10362}
}

• arXiv: 2407.10362