Nerrf Eval

nerrf-eval

NeRRF: 3D Reconstruction and View Synthesis for Transparent and Specular Objects with Neural Refractive-Reflective Fields — Chen et al. (2023) (arXiv:2309.13039, 2023)

What this evaluates

Evaluates the ability to reconstruct 3D geometry and synthesize novel views of transparent and specular objects from monocular RGB images and silhouettes. It probes physically accurate light path simulation, including refraction, reflection, and Fresnel effects, using a differentiable rendering framework.

Datasets

• Blender Synthetic Dataset — total 712; splits: train (400), test (312); repo https://github.com/dawning77/NeRRF

Metrics

• Chamfer Distance (CD) (primary) — range: other
  • Average minimum distance between predicted surface point cloud and ground truth surface point cloud. Lower is better.
• PSNR — range: other
  • Peak Signal-to-Noise Ratio in decibels, computed between predicted and ground truth RGB images. Higher is better.
• SSIM — range: [0, 1]
  • Structural Similarity Index Measure between predicted and ground truth images. Higher is better.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity between predicted and ground truth images. Lower is better.

Input / output format

Input: Monocular RGB rendering, segmentation mask of the object, camera pose, and material details (refractive/reflective type and index of refraction).

Output: Predicted surface point cloud or mesh for geometry; novel view RGB images for synthesis; estimated environment radiance map.

Scoring recipe

def compute_cd(pred_pts, gt_pts):
    d1 = min_dist(pred_pts, gt_pts).mean()
    d2 = min_dist(gt_pts, pred_pts).mean()
    return 0.5 * (d1 + d2)

def compute_psnr(pred_img, gt_img):
    mse = ((pred_img - gt_img) ** 2).mean()
    return 10 * log10(max_val**2 / mse)

def compute_ssim(pred_img, gt_img):
    return structural_similarity(pred_img, gt_img, channel_axis=-1)

def compute_lpips(pred_img, gt_img):
    return perceptual_loss(pred_img, gt_img)

Common pitfalls

• Baselines like IDR and PhySG do not model backgrounds, so novel view synthesis comparisons exclude background rendering.
• NeRO and NDR only handle reflective/glossy surfaces and cannot be evaluated on transparent objects.
• NeRFRO only handles reflective objects and is not evaluated on refractive scenes.

Evidence (verbatim from paper)

We conducted a comprehensive benchmark of our approach on the synthesized Blender dataset, utilizing a subset of the generated data for both training and testing. Specifically, we employed 50 images for training and 39 images for testing within each scene. During the geometry estimation stage, we assessed reconstruction quality using the Chamfer Distance (CD) between the predicted surface point cloud and the ground truth surface point cloud. For the environment estimation task, we employed standard image quality metrics, including PSNR, SSIM, and LPIPS for all quantitative assessments to gauge the performance.

Citation

@misc{chen2023nerrf,
  title={NeRRF: 3D Reconstruction and View Synthesis for Transparent and Specular Objects with Neural Refractive-Reflective Fields},
  author={Chen et al. (2023)},
  year={2023},
  note={arXiv:2309.13039}
}

• arXiv: 2309.13039