Urbanverse Eval

urbanverse-eval

UrbanVerse: Scaling Urban Simulation by Watching City-Tour Videos — Liu et al. (2025) (arXiv:2510.15018, 2025)

What this evaluates

Evaluates the fidelity of automated urban scene reconstruction from city-tour videos and the generalization capability of reinforcement learning navigation policies trained in these simulations. It measures how well generated scenes match real-world semantics and layouts, and how effectively policies transfer to unseen simulated and real-world environments.

Datasets

• KITTI-360 — total 45; splits: test (45)
• CraftBench — total 10; splits: test (10)
• AutoBench — total 10; splits: test (10)

Metrics

• success_rate (primary) — range: percent
  • Percentage of navigation attempts where the agent reaches the goal within a predefined distance tolerance without colliding. Calculated as (successful_trials / total_trials) * 100.
• route_completion — range: percent
  • Percentage of the total route distance successfully traversed by the agent before termination or collision.
• collision_times — range: count
  • Total number of collisions with obstacles or off-traversable surfaces during a navigation attempt.
• mAP25 — range: [0, 1]
  • 3D object detection mean Average Precision at an IoU threshold of 0.25, used to evaluate scene reconstruction fidelity against ground-truth annotations.

Input / output format

Input: RGB camera observations, relative position to the goal, and GPS-derived waypoints sampled every 5 meters projected into a local metric frame. No global map or semantic map is provided.

Output: Continuous or discrete navigation actions (e.g., velocity and steering commands) to move the agent toward the goal while avoiding collisions.

Scoring recipe

def compute_sr(predictions, gold):
    successful = 0
    for pred_pos, goal_info in zip(predictions, gold):
        dist = np.linalg.norm(np.array(pred_pos) - np.array(goal_info['target']))
        if dist <= goal_info['tolerance'] and not goal_info['collided']:
            successful += 1
    return (successful / len(gold)) * 100

Common pitfalls

• Evaluating on training scenes instead of the specified unseen AutoBench/CraftBench test sets, which inflates generalization scores.
• Assuming sim-to-real transfer requires domain adaptation or fine-tuning; the protocol explicitly tests zero-shot deployment on physical robots.
• Confusing layout diversity (number of unique city-tour videos) with intra-layout diversity (number of digital cousins), which have distinct scaling effects on policy performance.

Evidence (verbatim from paper)

Policy performance is measured by success rate (SR), route completion (RC), and collision times (CT), with all evaluations conducted in unseen environments from AutoBench and CraftBench.

Citation

@misc{liu2025urbanverse,
  title={UrbanVerse: Scaling Urban Simulation by Watching City-Tour Videos},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2510.15018}
}

• arXiv: 2510.15018