Ommo Eval

ommo-eval

A Large-Scale Outdoor Multi-modal Dataset and Benchmark for Novel View Synthesis and Implicit Scene Reconstruction — Lu et al. (2023) (arXiv:2301.06782, 2023)

What this evaluates

Evaluates the novel view synthesis and implicit scene reconstruction capabilities of NeRF-based methods on large-scale outdoor environments. It probes how well models handle diverse camera trajectories, varying lighting conditions, and different scene scales (e.g., buildings vs. cities).

Datasets

• OMMO — total 14700; splits: train (-1), test (-1)

Metrics

• PSNR (primary) — range: dB
  • Peak Signal-to-Noise Ratio. Measures pixel-level reconstruction error in decibels. Higher values indicate better fidelity.
• SSIM — range: [0, 1]
  • Structural Similarity Index. Measures perceived structural similarity between predicted and ground truth images. Ranges from 0 to 1, where 1 is identical.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity. Measures perceptual difference using deep network features. Lower values indicate higher perceptual similarity.

Input / output format

Input: Calibrated RGB images and camera poses for training views; novel camera poses for testing views.

Output: Synthesized RGB images for the requested novel test views.

Scoring recipe

def compute_metrics(pred_imgs, gt_imgs):
    mse = np.mean((pred_imgs - gt_imgs) ** 2)
    psnr = 10 * np.log10(255**2 / mse)
    ssim = compute_ssim(pred_imgs, gt_imgs)
    lpips = compute_lpips(pred_imgs, gt_imgs)
    return psnr, ssim, lpips

Common pitfalls

• NeRF-based methods frequently fail on scenes with irregular camera trajectories, performing significantly better on circular or line-based paths.
• Low-light/night scenes can artificially inflate PSNR/SSIM because networks output black areas to erase details, masking poor depth reconstruction.
• Large-scale scenes (e.g., cities) consistently yield lower metrics than small-scale subsets (e.g., buildings) due to limitations in implicit scene representation.

Evidence (verbatim from paper)

Evaluation Metrics. To evaluate the performance of each method, we use three common metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM)[[38]], and LPIPS[[45]] on novel view synthesis. Higher PSNR and SSIM mean better performancence, while a lower LPIPS means better.

Citation

@misc{lu2023ommo,
  title={A Large-Scale Outdoor Multi-modal Dataset and Benchmark for Novel View Synthesis and Implicit Scene Reconstruction},
  author={Lu et al. (2023)},
  year={2023},
  note={arXiv:2301.06782}
}

• arXiv: 2301.06782