Swim Eval

swim-eval

A New Dataset and Performance Benchmark for Real-time Spacecraft Segmentation in Onboard Flight Computers — Sam et al. (2025) (arXiv:2507.10775, 2025)

What this evaluates

Evaluates real-time instance segmentation performance of lightweight models under strict onboard hardware constraints, measuring inference speed, memory usage, and segmentation accuracy for spacecraft boundary localization.

Datasets

• SWiM — total 64000; splits: test (-1); repo https://github.com/RiceD2KLab/SWiM

Metrics

• RAM_footprint (primary) — range: MB
  • Peak RAM usage during full-precision inference on the target CPU.
• inference_time — range: seconds
  • Wall-clock time to process a single 640×640×3 image on the target hardware.
• model_parameters — range: M
  • Total number of trainable parameters in the model.
• segmentation_accuracy — range: [0, 1] | mm
  • Standard instance segmentation metrics (e.g., Dice score, Hausdorff distance) are used to evaluate mask quality, though specific formulas are not detailed in this section.

Input / output format

Input: RGB image resized to 640×640×3 pixels

Output: Bounding boxes and associated instance segmentation masks for detected spacecraft

Scoring recipe

import time
import psutil

def evaluate_model(model, image, ground_truth_masks):
    # Measure inference time
    start = time.perf_counter()
    predictions = model(image)
    elapsed = time.perf_counter() - start
    assert elapsed < 0.95, f"Inference time {elapsed:.3f}s exceeds 0.95s constraint"

    # Measure RAM footprint
    process = psutil.Process()
    ram_before = process.memory_info().rss
    _ = model(image)
    ram_after = process.memory_info().rss
    ram_used_mb = (ram_after - ram_before) / (1024 * 1024)
    assert ram_used_mb < 4096, f"RAM usage {ram_used_mb:.1f}MB exceeds 4GB constraint"

    # Compute accuracy (standard segmentation metrics)
    dice = compute_dice_score(predictions.masks, ground_truth_masks)
    hd = compute_hausdorff_distance(predictions.masks, ground_truth_masks)
    return {"inference_time": elapsed, "ram_footprint": ram_used_mb, "dice": dice, "hd": hd}

Common pitfalls

• Evaluating on GPU instead of the specified CPU-only Intel UP Board will drastically underestimate inference time and overstate real-world viability.
• Using larger YOLO variants (e.g., YOLOv8 small) exceeds the 0.95s constraint and 4GB RAM limit, violating the benchmark's hardware alignment.
• Resizing images to 640×640 may distort or obscure small spacecraft targets, artificially lowering segmentation accuracy metrics.

Evidence (verbatim from paper)

The segmentation algorithm must not only achieve high accuracy but also perform inference in real-time, ideally under 0.95s per image, to support spacecraft inspection during proximity operations in space. YOLOv8 Nano, with approximately 3.4M parameters and a 17 MB RAM footprint at full precision during inference, has already demonstrated successful deployment in space operations.

Citation

@misc{sam2025swim,
  title={A New Dataset and Performance Benchmark for Real-time Spacecraft Segmentation in Onboard Flight Computers},
  author={Sam et al. (2025)},
  year={2025},
  note={arXiv:2507.10775}
}

• arXiv: 2507.10775