Omni Recon Downstream Eval

omni-recon-downstream-eval

Omni-Recon: Harnessing Image-based Rendering for General-Purpose Neural Radiance Fields — Yonggan Fu et al. (arXiv:2403.11131, 2024)

What this evaluates

Evaluates a general-purpose NeRF framework on downstream 3D tasks, including real-time novel view synthesis, parameter-efficient 3D scene understanding, and text-guided 3D editing. It probes the model's ability to generalize to unseen scenes and adapt to various geometric and appearance tasks with minimal fine-tuning.

Datasets

• DTU — total ?; splits: test (-1)
• ScanNet — total ?; splits: train (-1), test (-1)

Metrics

• PSNR (primary) — range: other
  • Peak Signal-to-Noise Ratio computed between rendered novel-view images and ground truth images. Higher is better.
• FPS — range: other
  • Frames Per Second measured on an NVIDIA A5000 GPU to assess rendering speed.
• Sem. mIoU — range: percent
  • Mean Intersection over Union for semantic segmentation predictions on 3D scenes.
• Sem. Total Acc — range: percent
  • Overall accuracy of semantic segmentation predictions.
• Edge error — range: other
  • Error metric for edge detection (lower is better).
• Keypoint error — range: other
  • Error metric for 2D and 3D keypoint detection (lower is better).

Input / output format

Input: For rendering: novel view camera poses and source views to extract features and bake into a mesh. For understanding: 3D scene data (point clouds/meshes) from ScanNet with semantic labels, edges, and keypoints.

Output: Rendered novel view images and FPS; semantic segmentation maps, edge maps, and 3D keypoint coordinates.

Scoring recipe

def score(predictions, gold):
    psnr = compute_psnr(predictions.rendered_images, gold.images)
    fps = measure_fps(predictions.rendered_images, gold.cameras)
    mIoU = compute_mIoU(predictions.semantic_maps, gold.labels)
    acc = compute_accuracy(predictions.semantic_maps, gold.labels)
    edge_err = compute_error(predictions.edge_maps, gold.edges)
    kp_err = compute_error(predictions.keypoints, gold.keypoints)
    return {'PSNR': psnr, 'FPS': fps, 'Sem. mIoU': mIoU, 'Sem. Total Acc': acc, 'Edge error': edge_err, 'Keypoint error': kp_err}

Common pitfalls

• Finetuning time is capped at 5 minutes in the main table, but quality improves with longer finetuning (up to 5 min tested).
• FPS calculation assumes one-time feature extraction; actual throughput may be lower if feature extraction is included per frame.
• Evaluation on ScanNet follows a zero-shot generalizable setting on non-overlapping test scenes, not standard in-domain evaluation.

Evidence (verbatim from paper)

Our rendering pipeline can be swiftly finetuned to significantly boost the PSNR. For instance, with just a 10-second finetuning, our rendering pipeline can match the average rendering quality of the strongest baseline, ReTR [34]; With 1-minute/2-minute finetuning, it can surpass ReTR by a +2.75 and +3.36 PSNR improvement on average, respectively;

Citation

@misc{fu2024omnirecon,
  title={Omni-Recon: Harnessing Image-based Rendering for General-Purpose Neural Radiance Fields},
  author={Yonggan Fu et al.},
  year={2024},
  note={arXiv:2403.11131}
}

• arXiv: 2403.11131