Booster Eval

booster-eval

Booster: a Benchmark for Depth from Images of Specular and Transparent Surfaces — Zama Ramirez et al. (2023) (arXiv:2301.08245, 2023)

What this evaluates

Evaluates stereo and monocular depth/disparity estimation models on images containing specular and transparent surfaces, which violate standard non-Lambertian assumptions and cause significant performance degradation in existing networks.

Datasets

• Booster — total ?; splits: train (-1), test (-1)

Metrics

• bad-2 (primary) — range: percent
  • Percentage of pixels where the absolute disparity error exceeds 2 pixels.
• bad-4 — range: percent
  • Percentage of pixels where the absolute disparity error exceeds 4 pixels.
• bad-6 — range: percent
  • Percentage of pixels where the absolute disparity error exceeds 6 pixels.
• bad-8 — range: percent
  • Percentage of pixels where the absolute disparity error exceeds 8 pixels.
• MAE — range: other
  • Mean Absolute Error between predicted and ground-truth disparity values.
• RMSE — range: other
  • Root Mean Square Error between predicted and ground-truth disparity values.

Input / output format

Input: Stereo image pairs (left and right views) for balanced/unbalanced benchmarks; single image for monocular benchmark.

Output: Dense disparity maps (or depth maps) at the input resolution.

Scoring recipe

def compute_metrics(pred_disp, gt_disp, valid_mask):
    err = np.abs(pred_disp - gt_disp)
    bad2 = 100 * np.mean(err[valid_mask] > 2)
    bad4 = 100 * np.mean(err[valid_mask] > 4)
    bad6 = 100 * np.mean(err[valid_mask] > 6)
    bad8 = 100 * np.mean(err[valid_mask] > 8)
    mae = np.mean(err[valid_mask])
    rmse = np.sqrt(np.mean(err[valid_mask]**2))
    return {'bad-2': bad2, 'bad-4': bad4, 'bad-6': bad6, 'bad-8': bad8, 'MAE': mae, 'RMSE': rmse}

Common pitfalls

• Predictions are often upsampled via nearest-neighbor interpolation after processing at half/quarter resolution, which can inflate errors compared to downsampling ground truth.
• The benchmark evaluates on 'All' and 'Cons' (consistent) pixels, but occlusions are noted as not the main difficulty; material segmentation masks (Classes 0-3) are used to isolate specular/transparent regions.
• Disparity maps must be multiplied by the upsampling factor when comparing quarter-resolution predictions to full-resolution ground truth.

Evidence (verbatim from paper)

e.g., 33.07 vs 8.13 bad-2 for CREStereo on all-pixels in the Middlebury benchmark. This evidence confirms that high resolution is not the only challenge in our benchmark, but networks also struggle due to the presence of transparent and specular surfaces in our dataset.

Citation

@misc{zama_ramirez2023booster,
  title={Booster: a Benchmark for Depth from Images of Specular and Transparent Surfaces},
  author={Zama Ramirez et al. (2023)},
  year={2023},
  note={arXiv:2301.08245}
}

• arXiv: 2301.08245