Mdec Synspatches Eval

mdec-synspatches-eval

The Fourth Monocular Depth Estimation Challenge — Anton Obukhov et al. (2025) (arXiv:2504.17787, 2025)

What this evaluates

Evaluates monocular depth estimation models on their ability to predict accurate depth maps from single images. It specifically probes zero-shot generalization across diverse natural and indoor scenes using LiDAR-ground truth.

Datasets

• SYNS-Patches — total ?; splits: test (-1); repo https://github.com/toshas/mdec_benchmark

Metrics

• F-Score (primary) — range: percent
  • Harmonic mean of precision and recall for depth predictions, computed after aligning predictions to ground truth via least-squares or median scaling. Used as the primary ranking metric.
• AbsRel — range: percent
  • Absolute relative error between predicted and ground truth depth, averaged over valid pixels.
• MAE — range: other
  • Mean absolute error between predicted and ground truth depth.
• RMSE — range: other
  • Root mean squared error between predicted and ground truth depth.
• Acc-Edges — range: percent
  • Accuracy of predicted depth edges compared to ground truth edges.
• F-Edges — range: percent
  • F-Score computed specifically on depth edges.
• δ<1.25 — range: percent
  • Percentage of pixels where predicted depth is within a factor of 1.25 of ground truth.
• δ<1.25^3 — range: percent
  • Percentage of pixels where predicted depth is within a factor of 1.25^3 of ground truth.

Input / output format

Input: Single RGB image

Output: Predicted depth map (evaluated after alignment to ground truth)

Scoring recipe

def evaluate(pred_depth, gt_depth, align='least_squares'):
    if align == 'median':
        scale = np.median(gt_depth) / np.median(pred_depth)
        shift = 0.0
    else:
        # least-squares alignment with two degrees of freedom
        scale, shift = np.linalg.lstsq(...)
    pred_aligned = pred_depth * scale + shift
    f_score = compute_f1(pred_aligned, gt_depth)
    absrel = np.mean(np.abs(pred_aligned - gt_depth) / gt_depth)
    mae = np.mean(np.abs(pred_aligned - gt_depth))
    rmse = np.sqrt(np.mean((pred_aligned - gt_depth)**2))
    return {'F-Score': f_score, 'AbsRel': absrel, 'MAE': mae, 'RMSE': rmse}

Common pitfalls

• Forgetting to align predictions to ground truth before computing metrics; the protocol explicitly requires least-squares or median scaling.
• Confusing the primary ranking metric (F-Score) with image-based metrics like AbsRel or δ<1.25, which may rank teams differently.
• Assuming the benchmark uses metric depth; it explicitly uses affine-invariant/disparity-invariant evaluation.

Evidence (verbatim from paper)

Following the protocol from previous editions, submitted predictions were evaluated on the testing split of SYNS-Patches [[1], [96]], after being aligned to ground-truth depths according to median depth scaling or least-squares alignment, as requested by the participants. TableLABEL:tbl:res:results collects the results of this fourth edition of the challenge, ranking the submitted methods according to their F-Score performance.

Citation

@misc{obukhov2025mdec,
  title={The Fourth Monocular Depth Estimation Challenge},
  author={Anton Obukhov et al. (2025)},
  year={2025},
  note={arXiv:2504.17787}
}

• arXiv: 2504.17787