E3d Bench Eval

e3d-bench-eval

E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Models — Wenyan Cong et al. (2025) (arXiv:2506.01933, 2025)

What this evaluates

Evaluates the effectiveness, robustness, and inference efficiency of end-to-end 3D Geometric Foundation Models across sparse-view depth estimation, video depth estimation, and multi-view relative pose estimation. It probes models' ability to generalize across diverse domains including indoor, outdoor, aerial, and highly dynamic scenes under both normalized and metric-scale settings.

Datasets

• DTU — total ?; splits: test (-1)
• ETH3D — total ?; splits: test (-1)
• KITTI — total ?; splits: test (-1)
• Tanks and Temples — total ?; splits: test (-1)
• ScanNet — total ?; splits: test (-1)
• Bonn — total ?; splits: test (-1)
• TUM Dynamics — total ?; splits: test (-1)
• Sintel — total ?; splits: test (-1)
• PointOdyssey — total ?; splits: val (-1)
• Syndrome — total ?; splits: test (-1)
• CO3Dv2 — total ?; splits: test (-1)
• RealEstate10K — total ?; splits: test (-1)
• ScanNet-eval — total ?; splits: test (-1)
• KITTI Odometry — total ?; splits: test (-1)
• ADT — total ?; splits: test (-1)
• ACID — total ?; splits: test (-1)
• ULTRRA — total ?; splits: test (-1)

Metrics

• AbsRel (primary) — range: other
  • Measures the average relative error between model output and ground truth: mean(|pred - gt| / gt). Lower is better.
• δ < 1.03 — range: percent
  • Captures the percentage of pixels within 3% relative error (max(pred/gt, gt/pred) < 1.03). Higher is better.
• δ < 1.25 — range: percent
  • Captures the percentage of pixels within 25% relative error (max(pred/gt, gt/pred) < 1.25). Higher is better.
• ATE — range: other
  • Absolute Translation Error between predicted and ground-truth trajectories after Sim(3) alignment. Lower is better.
• RPE-trans — range: other
  • Relative Translation Error between consecutive predicted and ground-truth poses after Sim(3) alignment. Lower is better.
• RPE-rot — range: other
  • Relative Rotation Error between consecutive predicted and ground-truth poses after Sim(3) alignment. Lower is better.

Input / output format

Input: Sparse-view image pairs or triplets with minimal/no overlap for depth estimation; monocular video sequences for video depth estimation; multi-view image collections for relative pose estimation.

Output: Per-pixel depth maps (extracted from the z-coordinate of predicted point maps or via confidence-weighted averaging); relative camera pose trajectories (translation and rotation matrices).

Scoring recipe

def compute_depth_metrics(pred_depth, gt_depth, delta_threshold=1.03):
    abs_rel = np.mean(np.abs(pred_depth - gt_depth) / gt_depth)
    delta = np.mean(np.maximum(pred_depth / gt_depth, gt_depth / pred_depth) < delta_threshold)
    return abs_rel, delta

def compute_pose_metrics(pred_traj, gt_traj):
    pred_aligned = align_sim3(pred_traj, gt_traj)
    ate = np.mean(np.linalg.norm(pred_aligned - gt_traj, axis=-1))
    rpe_trans = np.mean(np.linalg.norm(np.diff(pred_aligned, axis=0) - np.diff(gt_traj, axis=0), axis=-1))
    rpe_rot = np.mean(np.arccos(np.clip(np.einsum('ij,ij->i', np.diff(pred_aligned, axis=0), np.diff(gt_traj, axis=0)), -1, 1)))
    return ate, rpe_trans, rpe_rot

Common pitfalls

• Confusing normalized vs. metric-scale evaluation protocols; normalized models require median depth scaling, while metric models should be evaluated on raw outputs or with an extra median-aligned setting.
• Failing to upsample all predictions to full resolution before computing metrics, which inflates error values.
• Not handling video padding or frame repetition for models that require fixed input lengths (e.g., padding sequences <16 frames for Geo4D, or using default 41-frame outputs for Aether).
• Applying a single Sim(3) alignment to datasets with separate coordinate systems (e.g., ULTRRA aerial/ground trajectories), which requires alternative metrics.

Evidence (verbatim from paper)

Evaluation Metrics: Two widely used metrics to evaluate depth estimation performance: Absolute Relative Error (AbsRel) measures the average relative error between model output and ground truth; the Inlier Ratio δ < 1.03 [36, 37] captures the percentage of pixels within 3% relative error.

Citation

@misc{cong2025e3dbench,
  title={E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Models},
  author={Wenyan Cong et al. (2025)},
  year={2025},
  note={arXiv:2506.01933}
}

• arXiv: 2506.01933