Robust Mvd Eval

robust-mvd-eval

A Benchmark and a Baseline for Robust Multi-view Depth Estimation — Schröppel et al. (2022) (arXiv:2209.06681, 2022)

What this evaluates

Evaluates multi-view depth estimation models on their ability to generalize across diverse domains, scales, and camera configurations. It specifically probes robustness to out-of-distribution cost volume statistics and tests absolute scale estimation without requiring scale alignment or depth range assumptions.

Datasets

• StaticThings3D — total 600; splits: test (60)
• BlendedMVS — total 70; splits: test (70)

Metrics

• depth error metrics (e.g., RMSE, AbsRel, δ1) (primary) — range: other
  • Standard depth estimation metrics comparing predicted depth maps against ground truth. Typically includes Root Mean Squared Error (RMSE), Absolute Relative Error (AbsRel), and threshold accuracy (δ1). Values are reported in Table A2 of the paper.

Input / output format

Input: Multi-view RGB images consisting of a key view and multiple source views, along with known camera poses. For StaticThings3D, the 5th view serves as the key with 4 preceding and 5 following source views. For BlendedMVS, 1 key frame is paired with 10 source views.

Output: Predicted depth map and predicted uncertainty map for the key view.

Scoring recipe

def compute_depth_metrics(pred_depth, gt_depth):
    # Handle invalid pixels (e.g., zero or NaN)
    valid = (gt_depth > 0) & (pred_depth > 0)
    pred = pred_depth[valid]
    gt = gt_depth[valid]
    
    # RMSE
    rmse = np.sqrt(np.mean((pred - gt) ** 2))
    # AbsRel
    abs_rel = np.mean(np.abs(pred - gt) / gt)
    # δ1 accuracy
    ratios = np.maximum(pred / gt, gt / pred)
    delta1 = np.mean(ratios < 1.25)
    
    return {'RMSE': rmse, 'AbsRel': abs_rel, 'δ1': delta1}

Common pitfalls

• Scale ambiguity: models often fail to estimate absolute depth without explicit scale alignment or depth range assumptions.
• Out-of-distribution cost volumes: existing models struggle when cost volume statistics differ significantly from training data.
• Qualitative-only real-world evaluation: the self-captured RealThings dataset is used only for qualitative demonstration, not quantitative benchmarking.

Evidence (verbatim from paper)

The StaticThings3D test split contains 600 sequences with 10 views each. We define one sample per sequence, using the fifth view as keyview plus 4 source views before and 5 after the keyview, resulting in 600 samples. To speed up evaluation, we use every 10th sample for the test set, resulting in a total of 60 samples. The BlendedMVS test set is based on the original validation split, which contains 7 scenes with a total of 915 frames. For each frame, a selection of 10 source views is provided by the authors. For our test set, we use 10 frames per scene as keyviews, resulting in a total of 70 samples. Results are provided in Tab. A2.

Citation

@misc{schrppel2022robustmvd,
  title={A Benchmark and a Baseline for Robust Multi-view Depth Estimation},
  author={Schröppel et al. (2022)},
  year={2022},
  note={arXiv:2209.06681}
}

• arXiv: 2209.06681