Surgcot Eval

surgcot-eval

SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark — Gui Wang et al. (2026) (arXiv:2604.20319, 2026)

What this evaluates

Evaluates multi-modal large language models on hierarchical spatiotemporal reasoning in surgical videos across five clinical dimensions: causal action ordering, cue-action alignment, affordance mapping, micro-transition localization, and anomaly onset tracking. The benchmark tests a model's ability to progressively narrow its focus from global video comprehension to fine-grained frame-level localization while maintaining logical consistency through a chain-of-thought protocol.

Datasets

• SurgCoT — total 2841; splits: test (-1); repo https://github.com/CVI-SZU/SurgCoT

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions (both main and sub-questions) per task, averaged across the five reasoning dimensions. Computed as (correct predictions / total questions) * 100.

Input / output format

Input: Surgical video clips (full videos for stage 1, bounded clips for stage 2, frames/patches for stage 3) paired with a clinically meaningful question, mutually exclusive multiple-choice options, domain knowledge priors, and video-grounded clues (temporal windows, spatial ROIs, or landmarks).

Output: A selected option from the provided candidates, optionally accompanied by a chain-of-thought rationale following the five-tuple protocol (Question→Option→Knowledge→Clue→Answer).

Scoring recipe

correct = 0
total = 0
for instance in dataset:
    pred = model.predict(instance.video, instance.question, instance.options, instance.knowledge, instance.clue)
    if pred == instance.answer:
        correct += 1
    total += 1
accuracy = (correct / total) * 100

Common pitfalls

• Models often fail to respect the progressive conditioning chain (A1→A2→A3), leading to logical inconsistencies when stage 2 or 3 ignores validated outputs from previous stages.
• Distractors are carefully designed to be clinically plausible, requiring fine-grained spatiotemporal grounding rather than relying on coarse visual recognition or general medical knowledge.
• The benchmark evaluates both main questions and sub-questions; ignoring sub-questions underestimates performance on fine-grained localization and temporal anchoring tasks.

Evidence (verbatim from paper)

Table 2: Evaluation of 10 MLLMs across five clinical reasoning tasks under progressive settings (BL→KE→FC) shows: 1) commercial models outperform open-source and medical-specialized counterparts, and 2) the five-tuple annotation protocol improves reasoning accuracy under both (KE) and (FC) settings.

Citation

@misc{wang2026surgcot,
  title={SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark},
  author={Gui Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.20319}
}

• arXiv: 2604.20319