Local Planner Benchmarking Eval

local-planner-benchmarking-eval

Benchmarking local motion planners for navigation of mobile manipulators — Tafnakaji et al. (2022) (arXiv:2211.01812, 2022)

What this evaluates

Compares local trajectory planners (DWB and TEB) for mobile manipulators by evaluating path smoothness, end-effector stability, trajectory deviation from a global path, and navigation accuracy/time in static and dynamic simulated environments.

Datasets

• Simulated Environments (Playground, Office, Warehouse) — total ?; splits: static (-1), dynamic (-1); repo https://github.com/sevag-tafnakaji/local_planner_benchmarking

Metrics

• $\mathbf{p}_{e}$ (end-effector stability) (primary) — range: other
  • Quantifies the difference between the expected trajectory of the end-effector and its real trajectory during navigation.
• $A_{\text{between}}$ (path deviation) — range: other
  • Measures the area between the global path and the local trajectory, indicating how much the robot deviates from the planned route.
• Smoothness — range: radians
  • Calculated as the angle between the current direction and the new direction at each point along the travelled path.
• $p_{\text{acc}}$ (accuracy) — range: other
  • Final position accuracy metric comparing the robot's end pose to the target.
• $T_{\text{taken}}$ (time) — range: seconds
  • Total time taken to complete the navigation task.

Input / output format

Input: Simulated environment map containing static and dynamic obstacles, a pre-computed global path, and robot kinematic constraints.

Output: A sequence of local trajectory waypoints (poses and velocities) generated by the planner to navigate from start to goal while avoiding obstacles.

Scoring recipe

def compute_metrics(traj_log, global_path, expected_ee_traj):
    angles = [angle_between(traj[i].dir, traj[i+1].dir) for i in range(len(traj)-1)]
    smoothness = mean(angles)
    deviation = integrate_area_between_paths(global_path, traj)
    ee_stability = mean(abs(expected_ee_traj - actual_ee_traj))
    accuracy = distance(traj[-1].pos, goal_pos)
    time_taken = traj[-1].timestamp - traj[0].timestamp
    return smoothness, deviation, ee_stability, accuracy, time_taken

Common pitfalls

• Dynamic obstacles can completely block DWB's small sampling window, causing total path planning failure.
• Measurement timing mismatches between base and arm sensors can artificially inflate z-axis stability metrics.
• High smoothness penalties in narrow pathways may reflect necessary navigation maneuvers rather than planner inefficiency.

Evidence (verbatim from paper)

The metric that quantifies the load’s stability ($\mathbf{p}_{e}$) can be seen in Fig. 3. Data on the average performance of algorithms can be found in Table I. We can see that, on average, DWB leads to more stable paths at the end-effector, suggesting that it provides more stable load carrying capabilities.

Citation

@misc{tafnakaji2022localplannerbenchmarking,
  title={Benchmarking local motion planners for navigation of mobile manipulators},
  author={Tafnakaji et al. (2022)},
  year={2022},
  note={arXiv:2211.01812}
}

• arXiv: 2211.01812