Stanford Orb Eval

stanford-orb-eval

Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmark — Kuang et al. (2023) (arXiv:2310.16044, 2023)

What this evaluates

Evaluates the ability of models to recover 3D geometry and surface material from images, and to synthesize novel views or relight objects in unseen real-world environments. It probes inverse rendering capabilities under natural, uncontrolled lighting conditions where ground-truth material is unavailable.

Datasets

• Stanford-ORB — total 2940; splits: train (60), test (10); repo https://github.com/StanfordORB/Stanford-ORB

Metrics

• Bidirectional Chamfer Distance (primary) — range: other (meters)
  • Computes the average minimum squared distance between points in the predicted mesh and the ground-truth scan, and vice versa. Measures geometric fidelity of the recovered 3D shape.
• Depth/Normal Map Error — range: [0, 1]
  • Compares predicted depth and normal maps against those generated from ground-truth 3D scans using L1 or angular error metrics.
• Rendered Image Reconstruction Error — range: [0, 1]
  • Renders the predicted geometry and material under ground-truth lighting (for relighting) or input lighting (for novel view), then compares the output image to the captured ground-truth image using standard image metrics (e.g., LPIPS, PSNR, SSIM).

Input / output format

Input: Multi-view masked images (or single image) of a real-world object, optionally accompanied by environment lighting maps and camera poses.

Output: Predicted 3D mesh (or depth/normal maps) and surface material properties (e.g., BRDF parameters).

Scoring recipe

def score_geometry(pred_mesh, gt_mesh):
    cd = bidirectional_chamfer_distance(pred_mesh, gt_mesh)
    depth_err = l1_error(pred_depth, gt_depth)
    normal_err = 1 - cosine_similarity(pred_normal, gt_normal)
    return cd, depth_err, normal_err

def score_image(pred_mesh, pred_mat, gt_light, gt_img, cam_pose):
    rendered = render(pred_mesh, pred_mat, gt_light, cam_pose)
    img_err = l1_or_lpips(rendered, gt_img)
    return img_err

Common pitfalls

• Material ground truth is unavailable in-the-wild, so evaluation relies on relighting quality as a proxy rather than direct material comparison.
• Real-world lighting changes during capture, requiring per-view environment maps solved from chrome ball reflections rather than a single static map.
• Automated segmentation masks (SAM/GrabCut) tend to underestimate object boundaries, requiring manual refinement for accurate evaluation.

Evidence (verbatim from paper)

We measure the quality of the geometry estimated from different methods by comparing the predicted depth maps and normal maps to those generated from the ground-truth scans, as well as directly computing the bidirectional Chamfer Distance between the predicted meshes and the scans. To evaluate the performance of a model, we render the object using the predicted geometry and materials and the ground-truth lighting of a new scene, and compare the rendered image with the captured ground-truth image from a given viewpoint.

Citation

@misc{kuang2023stanfordorb,
  title={Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmark},
  author={Kuang et al. (2023)},
  year={2023},
  note={arXiv:2310.16044}
}

• arXiv: 2310.16044