Mdec Syns Patches Eval

mdec-syns-patches-eval

The Second Monocular Depth Estimation Challenge — Spencer et al. (2023) (arXiv:2304.07051, 2023)

What this evaluates

Evaluates monocular depth estimation models across diverse real-world environments (natural, agricultural, urban, indoor) using high-quality LiDAR ground truth. Probes zero-shot generalization, boundary interpolation accuracy, and robustness to scene diversity and image artifacts.

Datasets

• SYNS-Patches — total ?; splits: test (-1)

Metrics

• F-Score (primary) — range: percent
  • Harmonic mean of precision and recall for depth estimation, typically computed at a specific threshold or for edge detection. Used as the primary ranking metric.
• F-Edges — range: percent
  • F-Score computed specifically on depth boundaries/edges to evaluate contour accuracy.
• MAE — range: other
  • Mean Absolute Error between predicted and ground truth depth values.
• RMSE — range: other
  • Root Mean Squared Error between predicted and ground truth depth values.
• AbsRel — range: other
  • Mean absolute relative error, calculated as the mean of |pred - gt| / gt across valid pixels.
• Acc-Edges — range: percent
  • Edge accuracy metric measuring the proportion of correctly predicted edge pixels.
• Comp-Edges — range: percent
  • Edge completion metric measuring the proportion of ground truth edges successfully recovered by the prediction.

Input / output format

Input: Monocular RGB image

Output: Predicted depth map (single-channel float tensor)

Scoring recipe

def compute_metrics(pred, gt):
    # Median alignment for fair comparison across supervision types
    pred_aligned = pred * (gt.median() / pred.median())
    
    # F-Score & F-Edges (edge-based)
    pred_edges = detect_edges(pred_aligned)
    gt_edges = detect_edges(gt)
    tp = (pred_edges & gt_edges).sum()
    precision = tp / pred_edges.sum() if pred_edges.sum() > 0 else 0
    recall = tp / gt_edges.sum() if gt_edges.sum() > 0 else 0
    f_score = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    
    # MAE, RMSE, AbsRel
    mae = np.mean(np.abs(pred_aligned - gt))
    rmse = np.sqrt(np.mean((pred_aligned - gt)**2))
    absrel = np.mean(np.abs(pred_aligned - gt) / gt)
    
    return f_score, f_edges, mae, rmse, absrel

Common pitfalls

• Models must be median-aligned to the ground truth before evaluation to ensure fair comparisons across different supervision types.
• Self-supervised methods trained exclusively on automotive data fail to generalize to indoor or non-automotive scenes.
• Thin structures (e.g., railings, branches) and transparent surfaces (e.g., glass) are consistently poorly estimated due to interpolation halos and lack of LiDAR/photometric constraints.

Evidence (verbatim from paper)

Participant submissions were evaluated on SYNS-Patches [1, 78]. As previously mentioned, this paper only discusses submissions that outperformed the baseline in any pointcloud-/image-based metric across the Overall dataset. Since both challenge phases ran independently and participants were responsible for generating the predictions, we cannot guarantee that the testing/Validation metrics used the same model. We therefore report results only for the test split. All methods were median aligned w.r.t. the ground-truth, regardless of the supervision used. This ensures that the evaluations are identical and comparisons are fair. Table 2 shows the overall performance for each submission across the whole dataset, as well as each category. Each subset is ordered using F-Score performance.

Citation

@misc{spencer2023mdec,
  title={The Second Monocular Depth Estimation Challenge},
  author={Spencer et al. (2023)},
  year={2023},
  note={arXiv:2304.07051}
}

• arXiv: 2304.07051