Oxford Spires Eval

oxford-spires-eval

The Oxford Spires Dataset: Benchmarking Large-Scale LiDAR-Visual Localisation, Reconstruction and Radiance Field Methods — Tao et al. (2024) (arXiv:2411.10546, 2024)

What this evaluates

Evaluates large-scale outdoor localization, 3D reconstruction, and novel-view synthesis using synchronized LiDAR, visual, and IMU data against millimetre-accurate TLS ground truth.

Datasets

• Oxford Spires Dataset — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• metric ground truth (primary) — range: meters
  • Computed by aligning predicted trajectories to the LiDAR/COLMAP trajectory using Umeyama's Sim(3) transformation, then measuring positional/orientational deviation against the TLS-derived ground truth (TUM format).
• Reconstruction accuracy — range: other
  • Compares predicted 3D point clouds or radiance fields against the registered TLS ground truth maps (1 cm resolution) to assess geometric fidelity.
• Novel view rendering/depth estimation — range: other
  • Evaluates rendered images or depth maps against ground truth depth images rendered from the TLS map using ground truth sensor trajectories.

Input / output format

Input: Synchronized RGB fisheye images (3 cameras at 20Hz), 3D LiDAR point clouds (10Hz), IMU measurements, COLMAP SfM outputs (camera parameters, poses, 3D points), and TLS ground truth maps.

Output: Estimated SE(3) trajectories, reconstructed 3D point clouds or radiance fields, and rendered depth/novel-view images.

Scoring recipe

# Load predicted poses and ground truth poses (TUM format)
# Estimate Sim(3) scale/alignment using Umeyama's method
scale_align = umeyama_alignment(pred_poses, gt_poses)
# Compute localization error on aligned poses
localization_error = compute_ate_rpe(pred_poses * scale_align, gt_poses)
# For reconstruction/rendering, compare predicted outputs against TLS ground truth
reconstruction_error = compute_chamfer_or_psnr(pred_reconstruction, tls_ground_truth)

Common pitfalls

• Vision-only methods (NeRF, 3DGS) often overfit to training poses and fail to generalize to out-of-sequence viewpoints.
• Metric scale ambiguity requires explicit Sim(3) alignment (Umeyama's method) before comparing to LiDAR/TLS ground truth.
• High-frequency image capture (20Hz) creates redundancy; methods must handle downsampled/aligned subsets (~1Hz) for SfM input.

Evidence (verbatim from paper)

Correcting the metric scale of vision-based 3D reconstructions produced by MVS and radiance field methods is necessary to enable comparison to the metric ground truth. To estimate the scale, we used Umeyama’s method to estimate a Sim(3) transformation between the LiDAR trajectory and a COLMAP trajectory, and the results are saved in evo_align_results.json.

Citation

@misc{tao2024oxfordspires,
  title={The Oxford Spires Dataset: Benchmarking Large-Scale LiDAR-Visual Localisation, Reconstruction and Radiance Field Methods},
  author={Tao et al. (2024)},
  year={2024},
  note={arXiv:2411.10546}
}

• arXiv: 2411.10546