Costnav Eval

costnav-eval

CostNav: A Navigation Benchmark for Cost-Aware Evaluation of Embodied Agents — Haebin Seong et al. (2025) (arXiv:2511.20216, 2025)

What this evaluates

Economic viability and cost-aware performance of embodied agents in urban sidewalk delivery navigation. It evaluates how technical metrics like collision rate and arrival success translate into real-world financial outcomes, including maintenance costs, energy usage, revenue, and break-even points.

Datasets

• CostNav Urban Sidewalk Navigation Simulation — total 100; splits: test (100); repo https://github.com/worv-ai/CostNav

Metrics

• Profit/run (primary) — range: dollar (can be negative)
  • Profit/run = R - C_run, where R is revenue per run and C_run is total operational cost per run. Negative values indicate commercial loss.
• Maintenance Cost ($/run) — range: dollar
  • c_shock × Collision Impulse × Collision Rate × C_hardware, where c_shock is a calibrated coefficient ($0.00001/N·s) and C_hardware is upfront robot cost.
• SLA Compliance (%) — range: percent [0, 100]
  • Percentage of episodes that terminate in 'Arrive' within the 600-second timeout threshold, directly measured from simulation.
• Revenue ($/run) — range: dollar
  • $3.49 × SLA Compliance (%), representing the expected earnings per delivery based on base fee and success rate.

Input / output format

Input: Physics-based simulation of urban sidewalks with varying pedestrian densities. The agent receives RGB-D camera feeds and compute module state to generate navigation actions.

Output: Navigation trajectory resulting in episode termination states (Collision, Arrive, Time-out) and logged physical metrics (collision impulse, power consumption, runtime).

Scoring recipe

arrival_rate = sum(1 for ep in episodes if ep.termination == 'Arrive') / 100
collision_rate = sum(1 for ep in episodes if ep.termination == 'Collision') / 100
avg_impulse = mean([ep.impulse for ep in episodes])
avg_power_w = mean([ep.power for ep in episodes])
runtime_hr = 1.0
c_shock = 0.00001
c_hardware = 11589
c_energy = (avg_power_w / 1000) * runtime_hr * 0.20
c_maint = c_shock * avg_impulse * collision_rate * c_hardware
c_run = c_energy + c_maint
sla = arrival_rate
revenue = 3.49 * sla
profit = revenue - c_run

Common pitfalls

• Scaling simulation results (20m, 6min) to real-world delivery parameters (6km, 1hr) assumes linear cost behavior, which may not hold for complex urban dynamics.
• Focusing solely on traditional success/arrival rate ignores that maintenance costs dominate (99.7% of run costs), making collision reduction the primary economic lever.
• Training collision costs are calculated using a calibrated shock coefficient, which introduces uncertainty if real-world maintenance pricing differs.

Evidence (verbatim from paper)

Profit/run = R - C_run = $1.501 - $31.51 = $ - 30.009 This represents the profit per delivery. Due to the high run cost overwhelming the revenue, the negative profit indicates that the method is not commercially viable at the current performance level.

Citation

@misc{seong2025costnav,
  title={CostNav: A Navigation Benchmark for Cost-Aware Evaluation of Embodied Agents},
  author={Haebin Seong et al. (2025)},
  year={2025},
  note={arXiv:2511.20216}
}

• arXiv: 2511.20216