Surgveo Eval

surgveo-eval

How Far Are Surgeons from Surgical World Models? A Pilot Study on Zero-shot Surgical Video Generation with Expert Assessment — Chen et al. (2025) (arXiv:2511.01775, 2025)

What this evaluates

Evaluates zero-shot surgical video generation models using a four-tiered Surgical Plausibility Pyramid. It probes the model's ability to maintain visual realism while correctly simulating domain-specific surgical causality, instrument handling, tissue feedback, and clinical intent over time.

Datasets

• SurgVeo benchmark — total ?; splits: laparoscopic (-1), neurosurgery (-1); repo https://github.com/franciszchen/SurgVeo

Metrics

• Visual Perceptual Plausibility (primary) — range: [1, 5]
  • Expert rating on a 1-5 scale assessing visual realism, texture quality, and lighting consistency of generated surgical frames.
• Instrument Operation Plausibility — range: [1, 5]
  • Expert rating on a 1-5 scale assessing the correctness of surgical tool manipulation, procedural actions, and anatomical targeting.
• Environment Feedback Plausibility — range: [1, 5]
  • Expert rating on a 1-5 scale assessing physical consistency of tissue response, fluid dynamics, and suction/irrigation behavior.
• Surgical Intent Plausibility — range: [1, 5]
  • Expert rating on a 1-5 scale assessing alignment with clinical goals, procedural logic, and step sequencing.

Input / output format

Input: Text prompts describing surgical procedures (Baseline or Stage-aware strategy) used to condition video generation models.

Output: Generated surgical videos (laparoscopic or neurosurgery tracks).

Scoring recipe

def compute_surgveo_scores(generated_videos, track):
    # Two domain experts independently rate each video on 1-5 scale
    expert1_ratings = [e1_score(v, track) for v in generated_videos]
    expert2_ratings = [e2_score(v, track) for v in generated_videos]
    # Average across experts for each SPP dimension
    final_scores = [(s1 + s2) / 2 for s1, s2 in zip(expert1_ratings, expert2_ratings)]
    return final_scores

Common pitfalls

• Assuming high visual fidelity implies correct surgical logic; the paper identifies a 'plausibility gap' where visual scores are high but causal/logical scores drop below 2.0.
• Expecting stage-aware prompting to significantly boost performance; results show it yields no consistent improvement over baseline prompts.
• Misclassifying error types; >93% of failures are high-level logical/surgical intent errors, not basic visual quality distortions.

Evidence (verbatim from paper)

Scores are reported on a 1-5 scale, where 5 is the best, and the standard deviation is calculated with the average score of two laparoscopic surgery experts. Our primary finding is a stark dichotomy in the performance of the generated surgical videos, clearly illustrated across both surgical types. The Veo-3 consistently excels at the base of the Surgical Plausibility Pyramid, achieving high scores in Visual Perceptual Plausibility.

Citation

@misc{chen2025surgveo,
  title={How Far Are Surgeons from Surgical World Models? A Pilot Study on Zero-shot Surgical Video Generation with Expert Assessment},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2511.01775}
}

• arXiv: 2511.01775