Geometry Preserving Depth Eval

geometry-preserving-depth-eval

Robust Geometry-Preserving Depth Estimation Using Differentiable Rendering — Chi Zhang et al. (arXiv:2309.09724, 2023)

What this evaluates

Evaluates monocular depth estimation models for their ability to produce geometry-preserving depth maps and accurate 3D point clouds without requiring explicit 3D annotations. It probes scale-and-shift recovery, generalization across indoor and outdoor domains, and consistency under differentiable rendering.

Datasets

• NYU V2 — total ?; splits: test (-1)
• ScanNet — total ?; splits: test (-1)
• KITTI — total ?; splits: test (-1)
• ETH3D — total ?; splits: test (-1)
• 2D3D — total ?; splits: test (-1)

Metrics

• AbsRel (primary) — range: [0, 1]
  • Absolute relative error computed as the mean of |D(i) - D*(i)| / D*(i) over all pixels M, where D is the scale-aligned prediction and D* is ground truth.
• δ1 — range: percent
  • Percentage of pixels where max(D(i)/D*(i), D*(i)/D(i)) < 1.25, measuring accuracy within a threshold.
• RMSE — range: other
  • Root Mean Square Error between the unprojected 3D point cloud (from aligned depth) and the ground-truth point cloud.

Input / output format

Input: RGB image resized to 384×384 pixels.

Output: Predicted depth map D (same spatial dimensions as input).

Scoring recipe

def compute_metrics(pred_depth, gt_depth, gt_pointcloud=None, pred_pointcloud=None):
    # Scale alignment per protocol
    s = np.median(gt_depth / pred_depth)
    pred_aligned = pred_depth * s
    
    # AbsRel
    absrel = np.mean(np.abs(pred_aligned - gt_depth) / gt_depth)
    
    # Delta1
    ratio = np.maximum(pred_aligned / gt_depth, gt_depth / pred_aligned)
    delta1 = np.mean(ratio < 1.25) * 100
    
    metrics = {'AbsRel': absrel, 'delta1': delta1}
    if gt_pointcloud is not None and pred_pointcloud is not None:
        metrics['RMSE'] = np.sqrt(np.mean((pred_pointcloud - gt_pointcloud)**2))
    return metrics

Common pitfalls

• Standard SSI loss produces scale-and-shift invariant outputs; failing to align scale using median(D*/D) before evaluation invalidates metrics.
• Omitting the shift term in SSI baselines causes severe geometric distortion, especially in outdoor scenes, leading to misleadingly poor point cloud reconstruction.
• Point cloud RMSE heavily depends on focal length selection; using an incorrect FOV drastically inflates error and should be optimized via consistency losses.

Evidence (verbatim from paper)

We evaluate our model on five benchmark datasets, including NYU V2, ScanNet, KITTI, ETH3D, and 2D3D. For evaluation of geometry-preserving depth estimation, we follow Leres to first align the scale of the predicted depth map D and the ground truth D* by multiplying the prediction with a factor s, which is computed by: s = median(D*/D). We use two common metrics to evaluate the accuracy: the absolute relative error (AbsRel): (1/|M|)∑|D(i)−D*(i)|/D*(i). and the percentage of pixels with δ1 = max(D(i)/D*(i), D*(i)/D(i)) < 1.25.

Citation

@misc{zhang2023robustgeometry,
  title={Robust Geometry-Preserving Depth Estimation Using Differentiable Rendering},
  author={Chi Zhang et al.},
  year={2023},
  note={arXiv:2309.09724}
}

• arXiv: 2309.09724