Robodepth Eval

robodepth-eval

RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptions — Kong et al. (2023) (arXiv:2310.15171, 2023)

What this evaluates

This benchmark evaluates the out-of-distribution robustness of monocular depth estimation (MDE) models when exposed to real-world corruptions such as weather changes, sensor failures, and data processing anomalies. It measures how much model performance degrades relative to a clean baseline across multiple corruption types and severity levels.

Datasets

• KITTI-C — total ?; splits: test (-1); repo https://github.com/ldkong1205/RoboDepth/blob/main/docs/DATA_PREPARE.md
• NYUDepth2-C — total ?; splits: test (-1); repo https://github.com/ldkong1205/RoboDepth/blob/main/docs/DATA_PREPARE.md
• KITTI-S — total ?; splits: test (-1); repo https://github.com/ldkong1205/RoboDepth/blob/main/docs/DATA_PREPARE.md

Metrics

• mCE (primary) — range: percent
  • Mean Corruption Error. Reported as a percentage (%) representing the average relative depth error degradation compared to a clean baseline model across all corruption types.
• mRR — range: percent
  • Mean Relative Robustness. Reported as a percentage (%) measuring the average relative robustness score compared to a clean baseline across all corruption types.
• mDEE — range: other
  • Mean Depth Estimation Error. The absolute depth estimation error averaged across the dataset.

Input / output format

Input: Corrupted images (monocular, stereo, or monocular+stereo pairs) from outdoor or indoor scenes.

Output: Predicted depth maps for each input image.

Scoring recipe

# For each model and corruption type:
error_clean = compute_depth_error(model, clean_images)
error_corrupted = compute_depth_error(model, corrupted_images)

# Calculate relative degradation
relative_errors = [error_corrupted[c] / error_clean[c] for c in corruption_types]
mCE = mean(relative_errors) * 100

# Calculate relative robustness
relative_robustness = [1 - (error_corrupted[c] - error_clean[c]) / error_clean[c] for c in corruption_types]
mRR = mean(relative_robustness) * 100

# Absolute error
mDEE = mean(error_corrupted)

Common pitfalls

• Failing to unify model configurations (backbones, data augmentations, post-processing) across candidates leads to unfair comparisons.
• Confusing mCE/mRR (relative degradation percentages) with raw absolute error values.
• Evaluating only on clean validation splits instead of the corrupted KITTI-C/NYUDepth2-C datasets misses the benchmark's core objective.

Evidence (verbatim from paper)

The mCE and mRR scores are given in percentage (%). Blocks from top to bottom: [1st]: The baseline MonoDepth2; [2nd]: Methods w/ monocular inputs; [3rd]: Methods w/ stereo inputs; [4th]: Methods w/ monocular + stereo inputs.

Citation

@misc{kong2023robodepth,
  title={RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptions},
  author={Kong et al. (2023)},
  year={2023},
  note={arXiv:2310.15171}
}

• arXiv: 2310.15171