Sparse Nvs Eval

sparse-nvs-eval

Where and How: Mitigating Confusion in Neural Radiance Fields from Sparse Inputs — Bao et al. (2023) (arXiv:2308.02908, 2023)

What this evaluates

Evaluates novel view synthesis performance from extremely sparse inputs (3 training views). It probes a model's ability to reconstruct 3D geometry and render photorealistic images for unseen camera poses without overfitting to the limited training data.

Datasets

• Realistic Synthetic 360° — total 280; splits: train (100), test (200)
• LLFF — total ?; splits: train (3), test (-1)

Metrics

• PSNR (primary) — range: other
  • Peak Signal-to-Noise Ratio in decibels, calculated as 10 * log10(MAX^2 / MSE) between rendered and ground truth images.
• SSIM — range: [0, 1]
  • Structural SIMilarity index measuring perceived structural changes between two images.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity using deep features to measure perceptual distance.

Input / output format

Input: 3 training RGB images with known camera poses (intrinsics and extrinsics) for a single scene.

Output: Rendered RGB images for each novel test viewpoint.

Scoring recipe

psnr_scores, ssim_scores, lpips_scores = [], [], []
for scene in datasets:
    train_imgs = scene.select_n_views(n=3)
    for test_view in scene.test_views:
        pred = model.render(train_imgs, test_view.pose)
        gt = test_view.image
        psnr_scores.append(calc_psnr(pred, gt))
        ssim_scores.append(calc_ssim(pred, gt))
        lpips_scores.append(calc_lpips(pred, gt))
return mean_std(psnr_scores), mean_std(ssim_scores), mean_std(lpips_scores)

Common pitfalls

• LPIPS is a lower-is-better metric, whereas PSNR and SSIM are higher-is-better.
• The N=3 sparse setting requires using the exact same randomly selected training viewpoints across all compared methods to ensure fairness.
• Background regions are often unobserved in sparse settings; foreground-only evaluation requires explicit background masking.

Evidence (verbatim from paper)

Metrics. We employ the standard image quality metrics, including Peak Signal to-Noise Ratio (PSNR) and Structural SIMilarity (SSIM) (Wang et al., 2004), to evaluate rendering quality for novel viewpoints. Additionally, we introduce learned perceptual image patch similarity (LPIPS) (Zhang et al., 2018) as a perceptual metric. For all metrics, we calculate the mean and standard deviation for comparative experiments. Following the experimental setup of InfoNeRF (Kim et al., 2022), under N=3 setting, we randomly select three viewpoints for training, and all comparison experiments use the same three viewpoints to ensure fairness.

Citation

@misc{bao2023wahnerf,
  title={Where and How: Mitigating Confusion in Neural Radiance Fields from Sparse Inputs},
  author={Bao et al. (2023)},
  year={2023},
  note={arXiv:2308.02908}
}

• arXiv: 2308.02908