Dynamic Urban View Synthesis Eval

dynamic-urban-view-synthesis-eval

Multi-Level Neural Scene Graphs for Dynamic Urban Environments — Tobias Fischer et al. (arXiv:2404.00168, 2024)

What this evaluates

Evaluates novel view synthesis and image reconstruction quality in dynamic urban environments containing fast-moving objects and varying environmental conditions. It measures how well a model can render unseen viewpoints and reconstruct training views while handling dynamic geometry and pose drift.

Datasets

• Argoverse 2 — total ?; splits: test (-1)
• KITTI — total ?; splits: test (-1)
• VKITTI2 — total ?; splits: test (-1)

Metrics

• PSNR (primary) — range: dB
  • Peak Signal-to-Noise Ratio in decibels. Computed as 10·log₁₀(MAX²/MSE), where MAX is the maximum possible pixel value (255 for 8-bit) and MSE is the mean squared error between predicted and ground truth images.
• SSIM — range: [0, 1]
  • Structural Similarity Index. Measures perceived change in structural information, luminance, and contrast between two images. Values range from -1 to 1, with 1 indicating identical images.
• LPIPS (AlexNet) — range: [0, 1]
  • Learned Perceptual Image Patch Similarity. Computes the L2 distance between deep features (AlexNet) of the reference and predicted images. Lower values indicate higher perceptual similarity.

Input / output format

Input: Multi-view image sequences (e.g., 7 ring-camera images per time step) with ground-truth camera poses and 3D bounding boxes for dynamic objects.

Output: Synthesized novel-view RGB images and depth maps.

Scoring recipe

def compute_metrics(pred_img, gt_img):
    mse = np.mean((pred_img - gt_img) ** 2)
    psnr = 10 * np.log10(255**2 / mse) if mse > 0 else 0
    ssim = structural_similarity(pred_img, gt_img, multichannel=True)
    lpips = perceptual_loss(pred_img, gt_img, model='alexnet')
    return psnr, ssim, lpips

Common pitfalls

• Pose drift heavily penalizes PSNR; hierarchical pose optimization is required to avoid misleadingly low scores.
• LPIPS directionality: lower values indicate better perceptual quality, opposite to PSNR and SSIM.
• Data splits are non-standard; Argoverse 2 uses every 10th frame held out, while KITTI/VKITTI2 follow the specific protocol from SUDS [61].

Evidence (verbatim from paper)

Following [61], we measure image synthesis quality with PSNR, SSIM [62], and LPIPS (AlexNet) [72]. To evaluate against competing methods on our proposed benchmark on Argoverse 2 [64], we hold out every 10th sample in uniform time intervals where a sample corresponds to seven ring-camera images. To compare our methods against prior art on KITTI [15] and VKITTI2 [4], we follow the experimental protocol and data splits in [61].

Citation

@misc{fischer2024multilevelneuralscenegraphs,
  title={Multi-Level Neural Scene Graphs for Dynamic Urban Environments},
  author={Tobias Fischer et al.},
  year={2024},
  note={arXiv:2404.00168}
}

• arXiv: 2404.00168