Extremenerf Eval

extremenerf-eval

Few-shot Neural Radiance Fields Under Unconstrained Illumination — SeokYeong Lee et al. (2023) (arXiv:2303.11728, 2023)

What this evaluates

Evaluates few-shot novel view synthesis and depth estimation under unconstrained, varying illumination. It probes a model's ability to maintain geometric consistency and produce photorealistic images when trained on only a few sparse views with different lighting conditions.

Datasets

• Phototourism F^3 — total ?; splits: (unstated)
• NeRF Extreme — total ?; splits: train (30), test (10)
• LLFF — total ?; splits: test (1)

Metrics

• SSIM (primary) — range: [0, 1]
  • Structural Similarity Index Measure. Computes luminance, contrast, and structure similarity between the synthesized and ground truth images. Values range from -1 to 1, where 1 indicates perfect similarity.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity. Measures perceptual difference by comparing deep feature activations (typically VGG or AlexNet) between images. Lower values indicate higher perceptual similarity.
• Abs Rel — range: other
  • Absolute Relative Error for depth estimation. Calculated as the mean of |d_pred - d_gt| / d_gt across valid pixels. Lower values indicate more accurate depth predictions.

Input / output format

Input: Sparse set of multi-view RGB images (typically 3 views) with known camera poses and intrinsics.

Output: Synthesized novel-view RGB images and corresponding depth maps.

Scoring recipe

def compute_metrics(pred_img, gt_img, pred_depth, gt_depth):
    ssim = calculate_ssim(pred_img, gt_img)
    lpips = calculate_lpips(pred_img, gt_img)
    valid_mask = gt_depth > 0
    abs_rel = np.mean(np.abs(pred_depth[valid_mask] - gt_depth[valid_mask]) / gt_depth[valid_mask])
    return {'SSIM': ssim, 'LPIPS': lpips, 'Abs Rel': abs_rel}

Common pitfalls

• PSNR is explicitly avoided because baselines fail at few-shot relighting, making it an unreliable metric for this setting.
• NeROIC diverges on frontal-facing scenes; only the geometry-only variant (NeROIC-Geom) is reported.
• Standard NeRF benchmarks like Phototourism are unsuitable for few-shot synthesis due to random view ordering; a curated frontal-facing subset is required.

Evidence (verbatim from paper)

For comparison, we used the mean SSIM, LPIPS metric of the synthesized image, and Abs Rel (Absolute Relative Error) of the synthesized depth map. Similar works have evaluated performance using PSNR after relighting to match the target illumination. However, our main aim is to highlight improved geometry details rather than relighting. Moreover, the baselines struggle with proper relighting in a few-shot setting, making PSNR unsuitable for evaluation.

Citation

@misc{lee2023extremenerf,
  title={Few-shot Neural Radiance Fields Under Unconstrained Illumination},
  author={SeokYeong Lee et al. (2023)},
  year={2023},
  note={arXiv:2303.11728}
}

• arXiv: 2303.11728