Panoptic Radiance Field Eval

panoptic-radiance-field-eval

In-Place Panoptic Radiance Field Segmentation with Perceptual Prior for 3D Scene Understanding — Li et al. (2024) (arXiv:2410.04529, 2024)

What this evaluates

Evaluates a NeRF-based method's ability to jointly reconstruct 3D scene geometry, appearance, and panoptic segmentation (semantic + instance) from multi-view images. It probes 3D consistency, boundary handling across indoor/outdoor scales, and robustness to pseudo-label noise via perceptual priors.

Datasets

• Replica — total ?; splits: test (-1)
• HyperSim — total ?; splits: test (-1)
• ScanNet — total ?; splits: test (-1)
• KITTI-360 — total ?; splits: test (-1)

Metrics

• PSNR — range: dB
  • Quantifies reconstructed luminance quality by measuring the difference between the rendered color image and the ground truth image.
• mIOU (primary) — range: percent
  • Evaluates semantic segmentation accuracy by calculating the intersection over union between the rendered semantic map and the ground truth semantic map, averaged across classes.
• PQ${}^{ ext{scene}}$ — range: percent
  • Assesses panoptic segmentation quality within the target scene by comparing the degree of alignment between the rendered semantic and instance maps with the supervised semantic and instance maps.
• SQ${}^{ ext{scene}}$ — range: percent
  • Measures segmentation accuracy of panoptic segmentation by evaluating the differences in segmentation between the rendered semantic and instance maps and the supervised semantic and instance maps.
• RQ${}^{ ext{scene}}$ — range: percent
  • Determines retrieval effectiveness of panoptic segmentation by comparing the retrieval discrepancies between the rendered semantic and instance maps and the supervised semantic and instance maps.

Input / output format

Input: Multi-view RGB images with corresponding camera poses, used to render 2D semantic and instance maps from a 3D implicit scene representation.

Output: Rendered 2D semantic map and 2D instance map per viewpoint, aligned with ground truth supervision.

Scoring recipe

def compute_miou(pred_semantic, gt_semantic, num_classes):
    ious = []
    for c in range(num_classes):
        pred_c = (pred_semantic == c)
        gt_c = (gt_semantic == c)
        intersection = np.sum(pred_c & gt_c)
        union = np.sum(pred_c | gt_c)
        iou = intersection / union if union > 0 else 1.0
        ious.append(iou)
    return np.mean(ious) * 100

Common pitfalls

• Resolution varies across datasets (512x512 for Replica/HyperSim, 256x256 for ScanNet, 1408x376 for KITTI-360), which directly impacts PSNR and mIOU scores.
• The 'Void' category is explicitly included in the semantic class count (22 indoor, 21 outdoor) and must be accounted for in IoU calculations.
• Scene-level metrics (PQ_scene, SQ_scene, RQ_scene) evaluate alignment between rendered and supervised maps per scene, not per-instance, which differs from standard panoptic benchmarks.

Evidence (verbatim from paper)

The proposed method is primarily assessed using the following evaluation metrics, where an upward arrow ($\uparrow$) signifies that higher values denote better performance, and vice versa: Peak Signal-to-Noise Ratio (PSNR$\uparrow$): This metric quantifies the quality of the reconstructed luminance by measuring the difference between the rendered color image and the ground truth image. Mean Intersection over Union (mIOU$\uparrow$): This metric evaluates the accuracy of semantic segmentation by calculating the intersection over union between the rendered semantic map and the ground truth semantic map.

Citation

@misc{li2024inplacepanoptic,
  title={In-Place Panoptic Radiance Field Segmentation with Perceptual Prior for 3D Scene Understanding},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2410.04529}
}

• arXiv: 2410.04529