Scared Eval

scared-eval

Stereo Correspondence and Reconstruction of Endoscopic Data Challenge — Allan et al. (2021) (arXiv:2101.01133, 2021)

What this evaluates

Evaluates stereo correspondence and depth estimation methods for endoscopic surgical scenes. It probes how accurately models can reconstruct quasi-dense depth maps from stereo image pairs captured with structured light on biological tissue.

Datasets

• SCARED — total ?; splits: test (10)

Metrics

• mean absolute error in mm (primary) — range: other
  • Mean absolute difference between predicted and ground truth depth at each pixel, averaged over valid pixels.

Input / output format

Input: Stereo image pairs (left and right views) from endoscopic scenes.

Output: Per-pixel depth map in millimeters.

Scoring recipe

def compute_mae_mm(pred_depth, gt_depth):
    valid_mask = gt_depth is not None
    diff = np.abs(pred_depth[valid_mask] - gt_depth[valid_mask])
    return np.mean(diff)

Common pitfalls

• Frames with less than 10% ground truth coverage are discarded entirely and not scored.
• Pixels without associated ground truth must be masked out before computing the mean error.
• Error is reported in absolute millimeters, not as a relative or normalized percentage.

Evidence (verbatim from paper)

The metric we use is the mean absolute error in mm of the depth measurement at each pixel. During the interpolation sequences, we mask the pixels which do not have associated ground truth so they are not considered in the error measurement and additionally we discard frames for which less than 10% of the frames have ground truth measurements.

Citation

@misc{allan2021scared,
  title={Stereo Correspondence and Reconstruction of Endoscopic Data Challenge},
  author={Allan et al. (2021)},
  year={2021},
  note={arXiv:2101.01133}
}

• arXiv: 2101.01133