Ophnet Eval

ophnet-eval

OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding — Ming Hu et al. (2024) (arXiv:2406.07471, 2024)

What this evaluates

Evaluates video understanding models on ophthalmic surgical workflows, including recognizing primary surgery types, classifying hierarchical surgical phases and operations, localizing phase boundaries in time, and anticipating upcoming surgical phases.

Datasets

• OphNet — total 2278; splits: train (1449), val (205), test (424)

Metrics

• Top-1 Accuracy (primary) — range: [0, 1]
  • Fraction of instances where the predicted class matches the ground truth label.
• Top-5 Accuracy — range: [0, 1]
  • Fraction of instances where the ground truth label appears in the model's top 5 predicted classes.
• mAP — range: [0, 1]
  • Mean Average Precision computed across classes at Intersection over Union thresholds [0.1, 0.3, 0.5, 0.7].
• Average mAP — range: [0, 1]
  • Arithmetic mean of mAP scores across the four IoU thresholds [0.1, 0.3, 0.5, 0.7].
• Top-1 Accuracy@ObservationRatio — range: [0, 1]
  • Fraction of correctly predicted next phases given a masked portion of the surgical sequence, evaluated at observation ratios [0.1, 0.3, 0.5, 0.7].
• Average Top-1 Accuracy — range: [0, 1]
  • Arithmetic mean of Top-1 Accuracy across the four observation ratios.

Input / output format

Input: Untrimmed surgical videos (presence recognition), trimmed video segments (phase/operation recognition), or concatenated RGB + optical flow feature embeddings (uniformly interpolated to 100 fixed-length frames) for localization/anticipation baselines.

Output: Class labels (surgery type, phase, or operation), temporal boundaries (start/end timestamps) for localization, or predicted next phase label for anticipation.

Scoring recipe

# Top-1/Top-5 Accuracy (Classification)
correct_top1 = sum(1 for p, g in zip(preds, gold) if p == g)
top1_acc = correct_top1 / len(gold)
correct_top5 = sum(1 for p, g in zip(preds, gold) if g in p[:5])
top5_acc = correct_top5 / len(gold)

# mAP / Average mAP (Localization)
aps = []
for iou_thresh in [0.1, 0.3, 0.5, 0.7]:
    aps.append(compute_ap(predictions, gold, iou_threshold=iou_thresh))
avg_mAP = sum(aps) / len(aps)

# Top-1 Accuracy@ObservationRatio (Anticipation)
accs = []
for obs_ratio in [0.1, 0.3, 0.5, 0.7]:
    masked_preds = model.predict(masked_sequence, ratio=obs_ratio)
    accs.append(sum(1 for p, g in zip(masked_preds, gold) if p == g) / len(gold))
avg_top1_acc = sum(accs) / len(accs)

Common pitfalls

• Variable video durations require uniform interpolation to exactly 100 fixed-length features before feeding to localization/anticipation models.
• Tags with fewer than 20 segments are filtered out, and 'Operation Gap'/'Invalid' labels are excluded from evaluation.
• Anticipation task uses random masking of phase sequences at specific observation ratios (0.1 to 0.9) rather than fixed temporal windows.

Evidence (verbatim from paper)

The results indicate that the TriDet model with a SwinViviT backbone outperforms other combinations, achieving the highest mAP scores across most IoU thresholds, with notable scores of 61.0% (IoU=0.1), 57.1% (IoU=0.3), 47.1% (IoU=0.5), and 33.1% (IoU=0.7), resulting in an average mAP of 50.4%.

Citation

@misc{hu2024ophnet,
  title={OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding},
  author={Ming Hu et al. (2024)},
  year={2024},
  note={arXiv:2406.07471}
}

• arXiv: 2406.07471