Carpatch Eval

carpatch-eval

CarPatch: A Synthetic Benchmark for Radiance Field Evaluation on Vehicle Components — Di Nucci et al. (2023) (arXiv:2307.12718, 2023)

What this evaluates

Evaluates the reconstruction quality of Neural Radiance Field (NeRF) models on synthetic vehicle components. It probes both 2D appearance fidelity and 3D geometric accuracy, specifically testing robustness to reflective surfaces, transparent materials, and varying numbers of training viewpoints.

Datasets

• CarPatch — total ?; splits: test (-1); repo https://github.com/davidedinuc/carpatch

Metrics

• PSNR (primary) — range: other
  • Peak Signal-to-Noise Ratio measuring pixel-wise reconstruction fidelity between predicted and ground truth RGB images.
• SSIM — range: [0, 1]
  • Structural Similarity Index measuring perceived structural changes between predicted and ground truth images.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity using deep feature distances to assess perceptual quality.
• D-RMSE — range: other
  • Depth Root Mean Squared Error: sqrt(mean((pred_depth - gt_depth)^2)). Computed only on pixels with positive depth in both maps.
• SN-RMSE — range: other
  • Surface Normal Root Mean Squared Error: sqrt(mean((arccos(n_pred) - arccos(n_gt))^2)). Measures angular error between predicted and ground truth surface normals.

Input / output format

Input: Per-view RGB images, camera poses, and ground-truth depth maps with component-wise semantic segmentation masks.

Output: Reconstructed RGB images, depth maps, and surface normals for each viewpoint.

Scoring recipe

# D-RMSE & SN-RMSE
valid_mask = (gt_depth > 0) & (pred_depth > 0)
d_rmse = np.sqrt(np.mean((gt_depth[valid_mask] - pred_depth[valid_mask])**2))
n_gt = normalize(depth_gradient(gt_depth))
n_pred = normalize(depth_gradient(pred_depth))
sn_rmse = np.sqrt(np.mean((np.arccos(np.clip(n_gt[valid_mask] * n_pred[valid_mask], -1, 1)))**2))
# Appearance metrics computed over bounding-box crops of each component

Common pitfalls

• D-RMSE and SN-RMSE must only be computed on pixels with positive depth in both ground truth and predicted maps to exclude background pixels (fixed depth 0).
• For component-level evaluation, appearance metrics (PSNR, SSIM, LPIPS) are computed over image crops delimited by bounding boxes around masks, not just foreground pixels, to prevent bias from unbalanced background/foreground ratios.

Evidence (verbatim from paper)

The effectiveness of the chosen methods has been assessed thanks to the typical perceptual metrics used in NeRF-based reconstruction tasks, namely PSNR, SSIM, and LPIPS. We suggest two supplementary depth-based metrics for the sole purpose of assessing the volume density. D-RMSE and SN-RMSE are computed only for those pixels with a positive depth value in both GT and predicted depth maps. This avoids computing depth estimation errors on background pixels (which have a fixed depth value of 0).

Citation

@misc{dinucci2023carpatch,
  title={CarPatch: A Synthetic Benchmark for Radiance Field Evaluation on Vehicle Components},
  author={Di Nucci et al. (2023)},
  year={2023},
  note={arXiv:2307.12718}
}

• arXiv: 2307.12718