qhjqhj00 mapless-navigation-eval
Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware — Tang et al. (2020) (arXiv:2003.01157, 2020)
What this evaluates
Evaluates a robot's ability to navigate to a goal in unknown environments using only local sensor data without a pre-built map. It probes the policy's generalization across varying obstacle densities, passage widths, and real-world conditions, as well as its energy efficiency on neuromorphic hardware.
Datasets
- Gazebo Training Environments — total ?; splits: train (-1)
- Gazebo Test Environment — total 200; splits: test (200)
- Real-world Office Environment — total 15; splits: test (15)
Metrics
success rate(primary) — range: percent- Percentage of test episodes that result in the robot successfully reaching the goal, as opposed to colliding with an obstacle or timing out after 1000 steps.
route distance— range: meters- Average Euclidean distance traveled by the robot along successful trajectories from start to goal.
route speed— range: m/s- Average speed maintained by the robot along successful trajectories.
energy cost per inference— range: other- Average power consumption (W) divided by inference speed (Inf/s) to yield energy cost per forward pass in microjoules.
Input / output format
Input: 18 laser range measurements (0.2-40 m) with 10-degree resolution, representing a front-facing 180-degree field of view.
Output: Continuous wheel speed commands constrained between 0.05 and 0.5 m/s.
Scoring recipe
outcomes = []
for start, goal in test_locations:
traj = run_episode(agent, start, goal, max_steps=1000)
outcomes.append(traj.outcome)
success_rate = sum(1 for o in outcomes if o == 'success') / len(outcomes)
route_distance = mean(len(traj) for traj in outcomes if traj.outcome == 'success')
route_speed = mean(traj.speed for traj in outcomes if traj.outcome == 'success')
energy_per_inf = avg_power_W / inference_speed_Inf_per_s * 1e6
Common pitfalls
- Evaluating on the same environments used for training instead of the held-out 20m x 20m test environment with different obstacle shapes and narrower passages.
- Comparing energy efficiency without accounting for hardware-specific power modes (e.g., TX2 MAXQ vs MAXN) or using idle power instead of dynamic inference power.
- Ignoring timeout episodes (1000 steps) when calculating success rates, which underestimates failure rates in complex environments.
Evidence (verbatim from paper)
We first compared the methods based on the rate of the three possible outcomes–success, collision, and timeout... we computed the average distance and speed corresponding to the successful routes taken by each method... The energy cost per inference was obtained by dividing the power consumed over 1 second with the number of inferences performed in that second.
Citation
@misc{tang2020sddpg,
title={Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware},
author={Tang et al. (2020)},
year={2020},
note={arXiv:2003.01157}
}
- arXiv: 2003.01157