Surglaivi Eval

surglaivi-eval

SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learning — Alejandra Perez et al. (arXiv:2509.10555, 2025)

What this evaluates

Evaluates zero-shot and linear-probe transferability of surgical vision-language models across laparoscopic and robotic video modalities. Probes hierarchical procedural understanding (phase, step, action) and object-centric recognition (tools, instrument-verb-target triplets) under varying temporal contexts and data regimes.

Datasets

• Cholec80 — total ?; splits: test (-1)
• AutoLaparo — total ?; splits: test (-1)
• GraSP — total ?; splits: test (-1)
• SARRARP50 — total ?; splits: test (-1)
• CholecT50 — total ?; splits: test (-1)

Metrics

• video-wise Accuracy — range: [0, 1]
  • Proportion of correctly classified video segments or frames, aggregated per video instance following Twinanda et al.
• video-wise F1-score (primary) — range: [0, 1]
  • Macro-averaged F1 score computed over video-wise predictions, balancing precision and recall across all surgical classes.
• mAP — range: [0, 1]
  • Mean Average Precision computed by treating cosine similarity scores as class probabilities for multi-label tool presence and triplet recognition tasks.

Input / output format

Input: A 16-frame (or 32-frame for linear probing) temporal window of a surgical video, encoded into a mean-pooled feature vector, compared against text embeddings of candidate classes.

Output: Top-similarity class label for classification tasks; similarity scores treated as class probabilities for multi-label tool presence and triplet recognition tasks.

Scoring recipe

def evaluate(predictions, gold, task_type):
    if task_type in ['phase', 'step', 'action']:
        acc = (predictions == gold).mean()
        f1 = f1_score(gold, predictions, average='macro')
        return {'accuracy': acc, 'f1': f1}
    else: # tool presence or triplets
        probs = predictions # similarity scores
        mAP = average_precision_score(gold, probs)
        return {'mAP': mAP}

Common pitfalls

• Using inconsistent inference prompts across models instead of each method's original prompts, which breaks fairness.
• Confusing frame-wise similarity scores with video-wise aggregated metrics (Accuracy/F1 vs mAP) without proper temporal pooling.
• Ignoring the impact of temporal window size (1, 8, 16, 32 frames) on performance, especially for fine-grained step recognition where 8 frames peak.

Evidence (verbatim from paper)

We report video-wise Accuracy and F1-score following Twinanda et al. [[22]]. For tool presence and CholecT50 triplets, we treat similarity scores as class probabilities and report mAP.

Citation

@misc{perez2025surglavi,
  title={SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learning},
  author={Alejandra Perez et al.},
  year={2025},
  note={arXiv:2509.10555}
}

• arXiv: 2509.10555