Endo Depth Robustness Eval

endo-depth-robustness-eval

Benchmarking Robustness of Endoscopic Depth Estimation with Synthetically Corrupted Data — An Wang et al. (2024) (arXiv:2409.16063, 2024)

What this evaluates

This benchmark evaluates the robustness of monocular depth estimation models when processing endoscopic images degraded by realistic surgical artifacts. It probes how well models maintain depth prediction accuracy and consistency under varying severities of illumination changes, optical blurs, visual obstructions, sensor noise, and compression artifacts.

Datasets

• Endoscopic Depth Estimation Dataset (Synthetically Corrupted) — total ?; splits: test (-1); repo https://github.com/lofrienger/EndoDepthBenchmark

Metrics

• DERS (primary) — range: other
  • DERS = (E / A) * exp(-R). E normalizes the average error across 4 metrics (AbsRel, SqRel, RMSE, LogRMSE) on corrupted images relative to clean images. A averages thresholded accuracy metrics (a1, a2, a3) across all corruption levels weighted by threshold strictness. R measures the standard deviation of all 7 metrics across corruption levels relative to clean performance, penalizing instability.

Input / output format

Input: Single-channel or RGB endoscopic images (clean and synthetically corrupted across 6 corruption types at 5 severity levels each).

Output: Per-pixel monocular depth prediction map.

Scoring recipe

def compute_ders(preds_clean, preds_corrupted, gt):
    # Compute 4 error metrics on clean (E0) and corrupted (Ej)
    E0 = [abs_rel, sq_rel, rmse, log_rmse](preds_clean, gt)
    Ej = [abs_rel, sq_rel, rmse, log_rmse](preds_corrupted, gt)
    E = sum(sum(Ej[i]) / (5 * E0[i]) for i in range(4))
    
    # Compute 3 accuracy metrics on clean (A0) and corrupted (Aj)
    A0 = [a1, a2, a3](preds_clean, gt)
    Aj = [a1, a2, a3](preds_corrupted, gt)
    A = sum((W[k] / 6) * sum(Aj[k][j] for j in range(6)) for k in range(3))
    
    # Compute robustness R across all 7 metrics
    M0 = E0 + A0
    Mj = [Ej[i] + Aj[i] for i in range(7)]
    R = (lambda_p / 7) * sum(sqrt(sum((Mj[i][j] - M0[i])**2 for j in range(1,6)) / 5) for i in range(7))
    
    return (E / A) * exp(-R)

Common pitfalls

• DERS requires normalizing corrupted error metrics against the clean image's error for the same metric, not against a fixed ground-truth scale or absolute values.
• The robustness component R penalizes performance variance across corruption severities rather than absolute degradation, so a consistently mediocre model may score better than a highly accurate but unstable one.
• Accuracy thresholds (a1, a2, a3) use multiplicative factors (<1.25, <1.25^2, <1.25^3) relative to ground truth depth, not absolute pixel differences.

Evidence (verbatim from paper)

To measure a model’s performance across various distortions common in endoscopic imaging, we present the Depth Estimation Robustness Score (DERS). DERS purposefully devised to combine three pivotal components—error, accuracy, and robustness—into a comprehensive composite index. ... Rendered as a whole, the Depth Estimation Robustness Score (DERS) integrates the aforementioned components, thus: DERS=rac{E}{A} imes e^{-R}.

Citation

@misc{wang2024endodepthbenchmark,
  title={Benchmarking Robustness of Endoscopic Depth Estimation with Synthetically Corrupted Data},
  author={An Wang et al. (2024)},
  year={2024},
  note={arXiv:2409.16063}
}

• arXiv: 2409.16063