Synthetic4relight Eval

synthetic4relight-eval

IRGS: Inter-Reflective Gaussian Splatting with 2D Gaussian Ray Tracing — Gu et al. (2024) (arXiv:2412.15867, 2024)

What this evaluates

Evaluates a 3D scene representation's capability for novel view synthesis, relighting, and inverse rendering (estimating diffuse albedo and roughness) from posed RGB images.

Datasets

• Synthetic4Relight — total ?; splits: test (-1)

Metrics

• PSNR (primary) — range: dB
  • Peak Signal-to-Noise Ratio, computed in decibels (dB) between rendered and ground truth images. Higher values indicate better reconstruction quality.
• SSIM — range: [0, 1]
  • Structural Similarity Index, measures perceptual structural similarity between images. Ranges from 0 to 1, where higher is better.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity, measures perceptual distance using deep network features. Lower values indicate better perceptual quality.
• MSE — range: [0, inf]
  • Mean Squared Error between estimated and ground truth material maps (e.g., roughness). Lower values indicate more accurate material estimation.

Input / output format

Input: A set of posed RGB images of a scene, along with object masks.

Output: Rendered RGB images for novel views and relighting scenarios, plus estimated material maps (albedo, roughness) and lighting parameters.

Scoring recipe

def compute_metrics(pred, gt):
    mse = np.mean((pred - gt) ** 2)
    psnr = 10 * np.log10(1.0 / mse)
    ssim = compute_ssim(pred, gt)
    lpips = compute_lpips(pred, gt)
    return psnr, ssim, lpips

# For material maps (e.g., roughness):
# mse_material = np.mean((pred_material - gt_material) ** 2)

Common pitfalls

• Relighting and novel view synthesis are evaluated under different lighting conditions; metrics must be computed separately for each task and not averaged together.
• Material estimation (albedo/roughness) requires disentangling appearance from lighting, so ground truth maps are compared directly rather than rendered appearance.
• LPIPS measures perceptual distance, meaning lower values indicate better quality, which is the opposite convention of PSNR and SSIM.

Evidence (verbatim from paper)

Table 1: Quantitative comparison on the Synthetic4Relight dataset[[43]]. A higher intensity of the red color signifies a better result.

| | Novel view synthesis | | | Relighting | | | Albedo | | | Roughness | Time | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | MSE↓ | Time (hours) |

Citation

@misc{gu2024irgs,
  title={IRGS: Inter-Reflective Gaussian Splatting with 2D Gaussian Ray Tracing},
  author={Gu et al. (2024)},
  year={2024},
  note={arXiv:2412.15867}
}

• arXiv: 2412.15867