Mtbbench Eval

mtbbench-eval

MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncology — Vasilev et al. (2025) (arXiv:2511.20490, 2025)

What this evaluates

Evaluates AI agents' ability to perform longitudinal, multimodal clinical decision-making in oncology. Agents must integrate evolving patient data across pathology, genomics, hematology, and imaging over multiple turns to answer diagnostic and prognostic questions, simulating molecular tumor board workflows.

Datasets

• MTBBench-Multimodal (HANCOCK subset) — total 390; splits: test (390)
• MTBBench-Longitudinal (MSK-CHORD subset) — total 183; splits: test (183)

Metrics

• accuracy (primary) — range: percent
  • Mean accuracy = (correct predictions / total questions) * 100. 95% confidence intervals are computed via bootstrap resampling (1,000 iterations) over patient-question pairs within each task.

Input / output format

Input: Multi-turn agentic interaction. At each turn t, the model receives a clinical query q_t and a set of modality-specific files F_t (e.g., H&E/IHC images, lab reports, genomic data, clinical timelines). The model may request subsets of F_t, which are only accessible within the current turn. Internal memory tracks reasoning history and previously accessed files. Longitudinal track includes an evolving timeline T_t.

Output: Multiple-choice or true/false answers to clinical questions, produced after iterative tool use and reasoning across turns.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

def bootstrap_ci(predictions, gold_labels, n=1000):
    import random
    accs = []
    for _ in range(n):
        idx = [random.randint(0, len(gold_labels)-1) for _ in range(len(gold_labels))]
        accs.append(compute_accuracy([predictions[i] for i in idx], [gold_labels[i] for i in idx]))
    accs.sort()
    return accs[int(0.025 * n)], accs[int(0.975 * n)]

Common pitfalls

• Files are non-persistent across turns; agents must actively re-request them rather than assuming static context.
• Static single-shot prompting is insufficient; the benchmark mandates interactive, multi-turn decision-making with selective tool invocation.
• Questions require reconciling conflicting evidence across heterogeneous modalities over time, not just isolated factual recall.

Evidence (verbatim from paper)

Table 2: Mean accuracy and 95% confidence intervals of various LLMs by task, estimated via bootstrap resampling. Each cell reports the model’s mean accuracy, with confidence intervals computed by resampling (with replacement) 1,000 times from the set of patient–question pairs within each task.

Citation

@misc{vasilev2025mtbbench,
  title={MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncology},
  author={Vasilev et al. (2025)},
  year={2025},
  note={arXiv:2511.20490}
}

• arXiv: 2511.20490