Mdec Eval

mdec-eval

The Monocular Depth Estimation Challenge — Spencer et al. (2022) (arXiv:2211.12174, 2022)

What this evaluates

Evaluates self-supervised monocular depth estimation models by measuring image-based accuracy and 3D pointcloud reconstruction quality. It probes the models' ability to generalize across diverse environments (urban, natural, agricultural, indoor) and highlights the impact of scale ambiguity and oversmoothing on relative object positioning.

Datasets

• SYNS-Patches — total ?; splits: test (-1); repo https://github.com/jspenmar/monodepth_benchmark

Metrics

• MAE — range: meters
  • Mean Absolute Error: average of the absolute differences between predicted and ground truth depth values across all pixels.
• RMSE — range: meters
  • Root Mean Square Error: square root of the average of squared differences between predicted and ground truth depth values.
• AbsRel — range: [0, 1]
  • Absolute Relative Error: mean of the absolute difference between predicted and ground truth depth divided by the ground truth depth across all pixels.
• F-Score (Edges) (primary) — range: [0, 1]
  • F1-score computed on pointcloud edge reconstruction, measuring the harmonic mean of precision and recall for detected edges in the 3D pointcloud output.

Input / output format

Input: Monocular RGB images from diverse environments (urban, natural, agricultural, indoor).

Output: Predicted depth maps and corresponding 3D pointclouds.

Scoring recipe

def compute_metrics(pred_depth, gt_depth):
    # Image-based metrics
    mae = np.mean(np.abs(pred_depth - gt_depth))
    rmse = np.sqrt(np.mean((pred_depth - gt_depth)**2))
    absrel = np.mean(np.abs(pred_depth - gt_depth) / gt_depth)
    # Pointcloud-based metric
    pred_edges = extract_edges_from_pointcloud(pred_depth)
    gt_edges = extract_edges_from_pointcloud(gt_depth)
    f_score_edges = f1_score(gt_edges, pred_edges)
    return {'MAE': mae, 'RMSE': rmse, 'AbsRel': absrel, 'F-Score (Edges)': f_score_edges}

Common pitfalls

• Over-optimizing for image-based metrics (MAE, AbsRel) while neglecting 3D pointcloud reconstruction quality (F-Score), which better reflects the true objective of depth estimation.
• Ignoring scale ambiguity, which makes absolute depth metrics difficult to compare across models without proper scaling or relative normalization.
• Oversmoothing predictions in highly textured regions, leading to poor edge completeness and interpolation artifacts between thin objects like railings or branches.

Evidence (verbatim from paper)

Table 3 show the performance of the participants' submissions on the SYNS-Patches test set. As seen, most submissions outperformed the baseline in traditional image-based metrics (MAE, RMSE, AbsRel) across all scene types. However, the baseline still achieved the best performance in both pointcloud reconstruction metrics (F-Score (Edges)). We believe this is due to the fact that most existing benchmarks report only image-based metrics. As such, novel contributions typically focus on improving performance on only these metrics. However, we believe pointcloud-based reconstruction metrics [39] are crucial to report, as they reflect the true objective of monocular depth estimation.

Citation

@misc{spencer2022monodepthchallenge,
  title={The Monocular Depth Estimation Challenge},
  author={Spencer et al. (2022)},
  year={2022},
  note={arXiv:2211.12174}
}

• arXiv: 2211.12174