Spacesense Bench Eval

spacesense-bench-eval

SpaceSense-Bench: A Large-Scale Multi-Modal Benchmark for Spacecraft Perception and Pose Estimation — Aodi Wu et al. (arXiv:2603.09320, 2026)

What this evaluates

Evaluates multi-modal spacecraft perception and pose estimation across 2D/3D segmentation, object detection, monocular depth estimation, and orientation estimation. Probes zero-shot generalization to unseen spacecraft configurations and robustness to long-tail class distributions and metallic surface reflections.

Datasets

• SpaceSense-Bench — total 136; splits: train (117), test (14); repo https://github.com/wuaodi/SpaceSense-Bench

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection-over-Union averaged across all 7 semantic classes. Computed as the mean of per-class IoU scores.
• aAcc — range: [0, 1]
  • Overall pixel accuracy, calculated as the total number of correctly classified pixels divided by the total number of pixels.
• mAP@0.5 — range: [0, 1]
  • Mean Average Precision at IoU threshold 0.5 across all classes.
• mAP@0.5:0.95 — range: [0, 1]
  • Mean Average Precision averaged over IoU thresholds from 0.5 to 0.95 in steps of 0.05.
• fwIoU — range: [0, 1]
  • Frequency-weighted IoU, where per-class IoU is weighted by the frequency of that class in the ground truth.
• AbsRel — range: [0, inf)
  • Absolute relative error between predicted and ground truth depth, averaged over valid pixels.
• Spearman rank correlation — range: [-1, 1]
  • Spearman's rho measuring the monotonic relationship between predicted and ground truth depth values.
• MAAE — range: [0, 180]
  • Mean Axis Angular Error, measuring the average angular difference between predicted and ground truth orientation axes.

Input / output format

Input: RGB images, depth maps, and/or LiDAR point clouds of spacecraft models rendered from various viewpoints. 2D tasks use 640×640 resolution images.

Output: Per-pixel or per-point semantic class labels, bounding boxes for detected objects, continuous depth values, and 6-DoF orientation angles/axes.

Scoring recipe

def compute_miou(pred_masks, gt_masks, num_classes=7):
    ious = []
    for c in range(num_classes):
        pred_c = (pred_masks == c)
        gt_c = (gt_masks == c)
        intersection = np.logical_and(pred_c, gt_c).sum()
        union = np.logical_or(pred_c, gt_c).sum()
        ious.append(intersection / union if union > 0 else 1.0)
    return np.mean(ious)
# For detection: compute AP@0.5 and AP@0.5:0.95 using standard COCO-style matching.
# For depth: AbsRel = mean(|d_pred - d_gt| / d_gt); Spearman = rank_correlation(d_pred, d_gt).
# For orientation: MAAE = mean(angular_distance(axis_pred, axis_gt)).

Common pitfalls

• Small components (e.g., thrusters, omni-antennas) occupy <0.2% of foreground pixels and suffer from severe long-tail performance drops.
• Zero-shot evaluation requires models to generalize to entirely unseen spacecraft geometries not seen during training.
• Depth estimation requires least-squares affine alignment on the satellite region before computing metrics like AbsRel.

Evidence (verbatim from paper)

For 2D semantic segmentation, we train FCN[11]* (ResNet-50), DeepLabV3+[5] (ResNet-50), SegFormer*[23]* (MiT-B3), and Mask2Former*[6]* (Swin-Base), all initialized from ImageNet pre-trained weights, and report mean Intersection-over-Union (mIoU) and overall pixel accuracy (aAcc). For object detection, we train YOLO26*[4]* at five scales (n/s/m/l/x) and report mAP@0.5 and mAP@0.5:0.95. For 3D point cloud segmentation, we train PMFNet*[27]* (ResNet-34) which fuses RGB and LiDAR, and report mIoU and frequency-weighted IoU (fwIoU). For monocular depth estimation, we evaluate Depth Anything V2*[24]* (ViT-S/B/L) in a zero-

Citation

@misc{wu2026spacesensebench,
  title={SpaceSense-Bench: A Large-Scale Multi-Modal Benchmark for Spacecraft Perception and Pose Estimation},
  author={Aodi Wu et al.},
  year={2026},
  note={arXiv:2603.09320}
}

• arXiv: 2603.09320