Rover Eval

rover-eval

ROVER: A Multi-Season Dataset for Visual SLAM — Schmidt et al. (2024) (arXiv:2412.02506, 2024)

What this evaluates

Evaluates the accuracy and robustness of visual-inertial SLAM systems across diverse outdoor environments, seasons, and lighting conditions. It probes long-term trajectory consistency, scale estimation, and environmental adaptability under challenging visual degradation.

Datasets

• ROVER — total 39; splits: test (39)

Metrics

• mATE (primary) — range: meters
  • Mean Absolute Trajectory Error. Computed as the average SE(3) alignment error between estimated and ground-truth poses across all frames in a sequence, then averaged over all locations and seasons.
• mRPE — range: meters
  • Mean Relative Pose Error. Computed as the average translation/rotation error between relative poses over a fixed interval, averaged over all locations and seasons.
• SR — range: percent
  • Success Rate. The percentage of sequences where the SLAM system maintains tracking without failure or excessive drift.

Input / output format

Input: Synchronized RGB images (mono, stereo, or RGBD) and IMU data from 39 sequences across 5 outdoor locations, 4 seasons, and 4 lighting conditions.

Output: Time-stamped camera pose trajectory (6-DoF) over the sequence duration.

Scoring recipe

def compute_rover_metrics(trajectories, ground_truths):
    ate_scores, rpe_scores, success_count = [], [], 0
    for est, gt in zip(trajectories, ground_truths):
        if not is_successful(est): success_count += 1
        ate_scores.append(compute_ate(est, gt))
        rpe_scores.append(compute_rpe(est, gt))
    return {
        'mATE': sum(ate_scores) / len(ate_scores),
        'mRPE': sum(rpe_scores) / len(rpe_scores),
        'SR': (success_count / len(trajectories)) * 100
    }

Common pitfalls

• Monocular SLAM requires scale alignment (e.g., Sim(3)) before ATE computation, otherwise errors are meaningless.
• SR is highly sensitive to tracking failures in low-light or dense vegetation, which can artificially deflate mATE if failed sequences are excluded.
• Internal vs. external IMU mounting significantly alters performance, so configurations must be reported explicitly.

Evidence (verbatim from paper)

For each SLAM configuration, we compute the ATE and RPE for every location (averaged across the four seasons) and then determine the overall mean ATE (mATE), mean RPE (mRPE), and SR across all locations.

Citation

@misc{schmidt2024rover,
  title={ROVER: A Multi-Season Dataset for Visual SLAM},
  author={Schmidt et al. (2024)},
  year={2024},
  note={arXiv:2412.02506}
}

• arXiv: 2412.02506