Cataract Lmm Eval

cataract-lmm-eval

Cataract-LMM: Large-Scale, Multi-Source, Multi-Task Benchmark for Deep Learning in Surgical Video Analysis — Ahmadi et al. (2025) (arXiv:2510.16371, 2025)

What this evaluates

Evaluates a model's ability to recognize and classify temporal surgical phases in cataract surgery videos, specifically testing classification accuracy and robustness to domain shift across different clinical centers.

Datasets

• Cataract-LMM Phase Recognition Subset — total 150; splits: train (80), val (26), test (44); repo https://github.com/MJAHMADEE/Cataract-LMM

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy computed over the 12 effective surgical phase classes (Viscoelastic and Anterior Chamber Flushing are merged into a single class).

Input / output format

Input: Video clips sampled at 4 frames per second (fps) from surgical procedures.

Output: Predicted surgical phase label for each input video clip.

Scoring recipe

def compute_metric(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Class imbalance is pronounced; Viscoelastic and Anterior Chamber Flushing are explicitly merged into a single class to mitigate this.
• Test set contains out-of-distribution data from a second hospital (Noor) to evaluate domain shift generalization.
• Videos are downsampled to 4 fps for evaluation, which may affect temporal resolution compared to original 30/60 fps recordings.

Evidence (verbatim from paper)

To rigorously assess model generalization, we partitioned the dataset based on the clinic of origin. The training set (80 videos) and validation set (26 videos) were drawn exclusively from the Farabi hospital. The test set consisted of 44 videos: 23 unseen videos from the Farabi hospital (in-distribution) and all 21 videos from the Noor hospital (out-of-distribution). To handle challenging classes, the visually similar and underrepresented phases, Viscoelastic and Anterior Chamber Flushing, were merged into a single class.

Citation

@misc{ahmadi2025cataractlmm,
  title={Cataract-LMM: Large-Scale, Multi-Source, Multi-Task Benchmark for Deep Learning in Surgical Video Analysis},
  author={Ahmadi et al. (2025)},
  year={2025},
  note={arXiv:2510.16371}
}

• arXiv: 2510.16371