Lasana Eval

lasana-eval

A benchmark for video-based laparoscopic skill analysis and assessment — Funke et al. (2026) (arXiv:2602.09927, 2026)

What this evaluates

Evaluates deep learning models on video-based laparoscopic surgical training tasks. It probes the model's ability to recognize task-specific procedural errors and predict structured global skill ratings from synchronized stereo video streams.

Datasets

• LASANA — total 1270; splits: train (944), val (121), test (205)

Metrics

• error_recognition (primary) — range: [0, 1]
  • Binary classification accuracy for each predefined task-specific error (e.g., object_dropped, cutting_imprecise). Each error is annotated as a binary yes/no label at the video level, and accuracy is computed per error type or averaged across all errors.
• skill_assessment — range: other
  • Mean Absolute Error (MAE) on the predicted Total Global Rating Score (GRS). The GRS is the sum of four Likert-scale aspects (depth perception, efficiency, bimanual dexterity, tissue handling) rated 1-5, yielding a raw range of 4-20.

Input / output format

Input: Synchronized stereo video recordings (left/right streams, 960×540 pixels, 20 fps, H.264 encoded) of one of four laparoscopic training tasks: peg transfer, circle cutting, balloon resection, or suture & knot.

Output: Per video: a binary vector indicating the presence/absence of each predefined task-specific error, and/or a continuous score representing the predicted Total Global Rating Score (GRS).

Scoring recipe

def score_error_recognition(preds, gold):
    correct = sum(1 for k in gold if preds[k] == gold[k])
    return correct / len(gold)

def score_skill_assessment(pred_grs, gold_grs):
    return abs(pred_grs - gold_grs)

# Aggregate over test set
total_error_acc = mean([score_error_recognition(v.pred_errors, v.gold_errors) for v in test_set])
total_skill_mae = mean([score_skill_assessment(v.pred_grs, v.gold_grs) for v in test_set])

Common pitfalls

• Splits are performed at the participant level, not video level; models must be evaluated on unseen individuals to test generalization.
• Ground truth skill ratings are normalized per rater before averaging to correct for Likert scale usage bias; raw rater scores should not be used directly.
• Failure criteria are task-specific (e.g., balloon opened on first cut for balloon resection); failed videos are excluded from skill rating but retained for error detection.

Evidence (verbatim from paper)

To provide a standardized benchmark for the development and evaluation of automatic video-based methods for skill assessment and error recognition, we define a data split for each task. Each split partitions the task recordings into training, validation, and test subsets in an approximate 75:10:15 ratio. The training set should be used for model parameter optimization, while the validation set can be utilized for hyperparameter optimization and monitoring model performance during training. The test set, by contrast, is reserved for the final, unbiased evaluation of any trained model.

Citation

@misc{funke2026lasana,
  title={A benchmark for video-based laparoscopic skill analysis and assessment},
  author={Funke et al. (2026)},
  year={2026},
  note={arXiv:2602.09927}
}

• arXiv: 2602.09927