Mobilitybench Eval

mobilitybench-eval

MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios — Zhiheng Song et al. (arXiv:2602.22638, 2026)

What this evaluates

Evaluates LLM-based route-planning agents on real-world mobility queries, probing their ability to handle multi-waypoint itineraries, preference-constrained routing, and multimodal travel. It measures how well agents understand instructions, decompose tasks, select tools, and produce valid, constraint-satisfying routes.

Datasets

• MobilityBench — total ?; splits: test (-1); repo https://github.com/AMAP-ML/MobilityBench

Metrics

• Final Pass Rate (FPR) (primary) — range: percent
  • Percentage of tasks where the agent's generated route fully satisfies all user constraints and passes validation.
• Delivery Rate (DR) — range: percent
  • Percentage of tasks where the agent successfully generates and delivers a route, regardless of full constraint satisfaction.

Input / output format

Input: Natural language user queries representing real-world mobility intents (e.g., multi-waypoint itineraries, preference-constrained routing, multimodal travel) from Amap.

Output: Structured route plans or sequences of tool calls and observations generated by the agent, expected to comply with the mapping API schema and satisfy task constraints.

Scoring recipe

def compute_metrics(predictions, golds):
    dr = sum(1 for p in predictions if p.is_delivered) / len(predictions)
    fpr = sum(1 for p in predictions if p.is_delivered and p.passes_constraints(golds)) / len(predictions)
    return {'DR': dr * 100, 'FPR': fpr * 100}

Common pitfalls

• Live mapping APIs are non-deterministic; evaluations must use a deterministic API-replay sandbox to ensure reproducibility.
• Preference-constrained routing tasks frequently cause hallucinations and trajectory deviations, especially under Plan-and-Execute frameworks.
• ReAct framework agents accumulate observation history, leading to significantly higher input token counts and inference latency compared to Plan-and-Execute.

Evidence (verbatim from paper)

Under the Plan-and-Execute framework, Claude-Opus-4.5 stands out as the strongest performer, achieved a Delivery Rate of 83.53% and a Final Pass Rate of 65.77%, both the highest among all evaluated models in this setting.

Citation

@misc{song2026mobilitybench,
  title={MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios},
  author={Zhiheng Song et al.},
  year={2026},
  note={arXiv:2602.22638}
}

• arXiv: 2602.22638