X Mobility Nav Eval

x-mobility-nav-eval

X-MOBILITY: End-To-End Generalizable Navigation via World Modeling — Liu et al. (2024) (arXiv:2410.17491, 2024)

What this evaluates

Evaluates an end-to-end navigation model's ability to predict robot dynamics and successfully navigate through structured and cluttered warehouse environments. It probes both open-loop trajectory and speed prediction accuracy, as well as closed-loop mission success, navigation efficiency, and motion smoothness in seen and out-of-distribution settings.

Datasets

• X-Mobility Warehouse Dataset — total 260000; splits: train (260000), test (-1)

Metrics

• mission success rate (SR) (primary) — range: percent
  • Percentage of closed-loop trials where the robot successfully reaches the goal within the environment.
• weighted trip time (WTT) — range: other
  • Navigation efficiency metric calculated by dividing the total trip time by the mission success rate.
• average absolute angular acceleration (AA) — range: other
  • Mean of the absolute angular acceleration values across a trial, used to indicate motion smoothness.
• A-MAE — range: other
  • Mean Absolute Error between predicted and ground truth angular speed in open-loop evaluation.
• L-MAE — range: other
  • Mean Absolute Error between predicted and ground truth linear speed in open-loop evaluation.
• P-MAE — range: other
  • Mean Absolute Error between predicted and ground truth path coordinates in open-loop evaluation.

Input / output format

Input: Sequential RGB images (R3×320×512), robot speed (R1), semantic label maps (R7×320×512), and optional route/path sequences. In closed-loop, the model receives step-wise observations and outputs action commands.

Output: Action command vector (R6) containing desired linear and angular velocities in x, y, and z dimensions.

Scoring recipe

def compute_metrics(predictions, ground_truth, env_traces):
    # Open-loop
    L_MAE = mean(abs(predictions.linear_speed - ground_truth.linear_speed))
    A_MAE = mean(abs(predictions.angular_speed - ground_truth.angular_speed))
    P_MAE = mean(abs(predictions.path - ground_truth.path))
    # Closed-loop
    successes = sum(1 for trace in env_traces if trace.reaches_goal())
    total_trials = len(env_traces)
    SR = successes / total_trials
    WTT = mean([trace.duration for trace in env_traces]) / SR
    AA = mean([mean(abs(trace.angular_acceleration)) for trace in env_traces])
    return {'SR': SR, 'WTT': WTT, 'AA': AA, 'L_MAE': L_MAE, 'A_MAE': A_MAE, 'P_MAE': P_MAE}

Common pitfalls

• Open-loop MAE metrics are evaluated on a held-out test split of the training dataset, not on the closed-loop benchmark scenarios.
• WTT is defined as trip time divided by success rate, not simply average time-to-goal, which can mislead efficiency comparisons when success rates vary.
• Closed-loop evaluation uses 100 runs total (5 trials per scenario), so reporting variance or confidence intervals alongside means is critical for robust comparison.

Evidence (verbatim from paper)

For the open-loop evaluation, we measured the Mean Absolute Error (MAE) for linear speed (L-MAE), angular speed (A-MAE), and path prediction (P-MAE) using the test split of the training dataset. In the closed-loop evaluation, we built a navigation benchmark suite with 10 warehouse scenarios of varying difficulty... We tracked three key metrics: 1) mission success rate (SR), 2) weighted trip time (WTT), which measures navigation efficiency by dividing trip time by success rate, and 3) average absolute angular acceleration (AA) as an indicator of motion smoothness.

Citation

@misc{liu2024xmobility,
  title={X-MOBILITY: End-To-End Generalizable Navigation via World Modeling},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2410.17491}
}

• arXiv: 2410.17491