Posenet Eval

posenet-eval

PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization — Kendall et al. (2015) (arXiv:1505.07427, 2015)

What this evaluates

Evaluates a model's ability to estimate 6-DOF camera pose (translation and rotation) from a single monocular image across indoor and outdoor environments. It probes the network's robustness to challenging conditions like motion blur, low light, and dynamic objects, as well as its generalization to unseen scenes and varying training baselines.

Datasets

• 7 Scenes — total ?; splits: test (-1)
• Cambridge Landmarks — total ?; splits: test (-1)

Metrics

• localization error (primary) — range: other
  • Cumulative histogram of translation error (in meters) and rotation error (in degrees) between the predicted and ground truth 6-DOF camera poses. Lower values indicate better performance.

Input / output format

Input: Single RGB image

Output: 6-DOF camera pose (3D translation vector and 3D rotation matrix or quaternion)

Scoring recipe

def compute_pose_error(pred_pose, gt_pose):
    # pred_pose, gt_pose are 4x4 SE(3) matrices or [t, R]
    t_err = np.linalg.norm(pred_pose[:3] - gt_pose[:3])
    R_err = pred_pose[:3, :3].T @ gt_pose[:3, :3]
    rot_err = np.arccos(np.clip((np.trace(R_err) - 1) / 2, -1, 1))
    return t_err, rot_err

# Aggregate over test set
t_errors = [compute_pose_error(p, g)[0] for p, g in zip(preds, gold)]
rot_errors = [compute_pose_error(p, g)[1] for p, g in zip(preds, gold)]
# Plot cumulative distribution functions for t_errors and rot_errors

Common pitfalls

• Ground truth poses are generated via Structure-from-Motion (SfM), which can be inaccurate or fail entirely in textureless, ambiguous, or highly dynamic regions.
• For extreme conditions like fog, rain, or night, SfM fails to produce ground truth, so quantitative metrics are unavailable and accuracy is assessed visually rather than numerically.
• The model is evaluated on cumulative histograms rather than single-point metrics, so reporting a single number without specifying the percentile or threshold can be misleading.

Evidence (verbatim from paper)

Fig.[6] shows cumulative histograms of localization error for two indoor and two outdoor scenes. We note that although the SCoRe forest is generally more accurate, it requires depth information, and uses higher-resolution imagery. The indoor dataset contains many ambiguous and textureless features which make relocalization without this depth modality extremely difficult.

Citation

@misc{kendall2015posenet,
  title={PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization},
  author={Kendall et al. (2015)},
  year={2015},
  note={arXiv:1505.07427}
}

• arXiv: 1505.07427