Spacecraft Pose Estimation Eval

spacecraft-pose-estimation-eval

Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering — Proença et al. (2019) (arXiv:1907.04298, 2019)

What this evaluates

Evaluates a model's ability to estimate the 6D pose (position and orientation) of non-cooperative spacecraft from monocular images. It probes generalization from photorealistic synthetic data to real space imagery and tests robustness to perceptual aliasing and orientation ambiguity.

Datasets

• URSO — total 5000; splits: train (4000), val (500), test (500)
• SPEED — total ?; splits: train (12005), test (3298)

Metrics

• mean absolute location error — range: other
  • L2 distance between predicted and ground-truth translation vectors.
• mean angular error — range: other
  • Geodesic distance (in degrees) between predicted and ground-truth orientation rotations.
• ESA Error (primary) — range: other
  • Sum of the mean relative location error and the mean angular error, as used by the official ESA challenge server.

Input / output format

Input: Grayscale images of spacecraft (1920x1200 px or resized to half), paired with ground truth 6D pose labels (position and orientation/Euler angles).

Output: Predicted 6D pose (translation vector and orientation, typically as Euler angles or rotation matrix) per image.

Scoring recipe

def compute_metrics(preds, golds):
    loc_errs, ang_errs, rel_loc_errs = [], [], []
    for pred, gt in zip(preds, golds):
        t_pred, t_gt = pred[:3], gt[:3]
        r_pred, r_gt = pred[3:], gt[3:]
        loc_err = np.linalg.norm(t_pred - t_gt)
        ang_err = angular_distance(r_pred, r_gt)
        rel_loc_err = loc_err / np.linalg.norm(t_gt)
        loc_errs.append(loc_err)
        ang_errs.append(ang_err)
        rel_loc_errs.append(rel_loc_err)
    return np.mean(loc_errs), np.mean(ang_errs), np.mean(rel_loc_errs) + np.mean(ang_errs)

Common pitfalls

• Test set ground-truth labels are withheld; evaluation must be performed via the official ESA submission server, not locally.
• Hard classification of orientation bins fails due to perceptual aliasing; soft classification or direct regression is required.
• Models trained purely on synthetic data overfit quickly; camera rotation augmentation is mandatory for real-world generalization.

Evidence (verbatim from paper)

Performance is reported as the mean absolute location error, the mean angular error and also the metric used by the ESA challenge server, referred to as ESA Error, which is the sum of the mean relative location error, as in (1), and the mean angular error.

Citation

@misc{proenca2019deep,
  title={Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering},
  author={Proença et al. (2019)},
  year={2019},
  note={arXiv:1907.04298}
}

• arXiv: 1907.04298