Tanks Temples Truck Eval

tanks-temples-truck-eval

Learning Neural Radiance Fields from Multi-View Geometry — Orsingher et al. (2022) (arXiv:2210.13041, 2022)

What this evaluates

Evaluates novel view synthesis quality and 3D geometric reconstruction accuracy of NeRF variants on a single outdoor scene.

Datasets

• Tanks & Temples (Truck scene) — total 250; splits: train (225), test (25)

Metrics

• PSNR — range: other
  • Peak Signal-to-Noise Ratio computed as -10 * log10(MSE(I, I_hat)), where MSE is the mean squared error between rendered and ground truth images. Higher is better.
• SSIM — range: [0, 1]
  • Structural Similarity Index Measure that evaluates perceived changes in structural information, luminance, contrast, and structure between rendered and ground truth images. Higher is better.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity measuring the L2 distance between feature activations of a pre-trained AlexNet for corresponding image patches. Lower is better.
• CD (primary) — range: other
  • Chamfer Distance computed as 1/|P| sum_{x in P} min_{y in P_hat} ||x-y||^2 + 1/|P_hat| sum_{y in P_hat} min_{x in P} ||x-y||^2, measuring geometric fidelity between the extracted mesh vertices and the laser scanner point cloud. Lower is better.

Input / output format

Input: Camera poses and training images for novel view synthesis; density and color fields for mesh extraction.

Output: Rendered RGB images for held-out test poses; triangular mesh vertices extracted via marching cubes.

Scoring recipe

def compute_metrics(rendered_img, gt_img, mesh_verts, gt_pc):
    psnr = -10 * np.log10(np.mean((rendered_img - gt_img)**2))
    ssim = compute_ssim(rendered_img, gt_img)
    lpips = compute_lpips(rendered_img, gt_img)
    cd = (1/len(gt_pc) * np.min(np.linalg.norm(gt_pc[:, None] - mesh_verts[None, :], axis=2)**2, axis=1).mean() +
          1/len(mesh_verts) * np.min(np.linalg.norm(mesh_verts[:, None] - gt_pc[None, :], axis=2)**2, axis=1).mean())
    return psnr, ssim, lpips, cd

Common pitfalls

• PSNR, SSIM, and LPIPS evaluate novel view synthesis, while Chamfer Distance evaluates 3D mesh geometry; they measure fundamentally different aspects and should not be conflated.
• Chamfer Distance is computed between the extracted mesh vertices (via marching cubes) and the laser scanner point cloud, not the raw NeRF point cloud or density field.
• Evaluation is performed on a single scene (Truck from Tanks & Temples), so results do not generalize to other datasets or indoor scenes.

Evidence (verbatim from paper)

Consistently with existing literature [[18], [21], [3]], three metrics are used to evaluate the quality of novel views: The Peak Signal-to-Noise Ratio (PSNR) is defined as follows: PSNR = -10/log(10) * MSE(I, I_hat)... Moreover, we want to quantify the geometric results to prove that MVG-NeRF generates better 3D models. To this end, the Chamfer distance between the point cloud from the laser scanner and the mesh vertices after marching cubes is computed (lower is better): CD(P, P_hat) = 1/|P| sum_{x in P} min_{y in P_hat} ||x-y||^2 + 1/|P_hat| sum_{y in P_hat} min_{x in P} ||x-y||^2

Citation

@misc{orsingher2022mvgnerf,
  title={Learning Neural Radiance Fields from Multi-View Geometry},
  author={Orsingher et al. (2022)},
  year={2022},
  note={arXiv:2210.13041}
}

• arXiv: 2210.13041