Target Bench Eval

target-bench-eval

Target-Bench: Can World Models Achieve Mapless Path Planning with Semantic Targets? — Wang et al. (2025) (arXiv:2511.17792, 2025)

What this evaluates

Evaluates whether world models can perform mapless path planning toward semantic targets in real-world environments. It probes spatio-temporal consistency, trajectory accuracy, and the ability to reason about explicit versus implicit goals without prior map information.

Datasets

• Target-Bench — total 450; splits: train (325), test (-1)

Metrics

• FDE — range: meters
  • Final Displacement Error: Euclidean distance between the predicted and ground-truth final positions. Exact threshold for Miss Rate is defined in Sec. 3.2.2.
• ADE — range: meters
  • Average Displacement Error: Mean Euclidean distance between predicted and ground-truth positions across the trajectory.
• MR — range: percent
  • Miss Rate: Percentage of trajectories where the final position exceeds a predefined error threshold.
• SE — range: [0, 1]
  • Soft Endpoint: Accuracy/probability score measuring how closely the predicted endpoint matches the target.
• AC — range: [0, 1]
  • Approach Consistency: Measures directional alignment between the predicted trajectory and the ground-truth path.
• WO (primary) — range: [0, 1]
  • Weighted Overall score: A composite metric aggregating FDE, ADE, MR, SE, and AC to provide a single performance indicator.

Input / output format

Input: Robot-collected video sequences (720p/1080p, 5–10s duration) paired with explicit or implicit semantic target descriptions.

Output: Generated video sequences depicting the predicted trajectory toward the target.

Scoring recipe

def compute_metrics(pred_video, gt_video, tool='VGGT'):
    pred_traj = reconstruct_trajectory(pred_video, tool)
    gt_traj = extract_trajectory(gt_video)
    fde = distance(pred_traj[-1], gt_traj[-1])
    ade = mean([distance(p, g) for p, g in zip(pred_traj, gt_traj)])
    mr = 1 if fde > threshold else 0
    se = endpoint_similarity(pred_traj[-1], gt_traj[-1])
    ac = directional_alignment(pred_traj, gt_traj)
    wo = weighted_average([fde, ade, mr, se, ac])
    return {'FDE': fde, 'ADE': ade, 'MR': mr, 'SE': se, 'AC': ac, 'WO': wo}

Common pitfalls

• Scores vary significantly depending on the spatio-temporal reconstruction tool used (VGGT vs. SpaTracker vs. ViPE).
• Performance is highly sensitive to the planning horizon length; shorter horizons (4s) consistently yield higher scores than longer ones (8s).
• Explicit and implicit targets produce similar results, but model-specific fluctuations can mislead comparisons if not averaged across both.

Evidence (verbatim from paper)

Among all evaluated off-the-shelf models, Wan2.2-Flash achieves the best overall performance with a weighted overall score of 0.299. Specifically, it obtains the lowest errors in FDE (1.362m), ADE (1.005m), and MR (38.75%), while achieving the highest SE (0.292).

Citation

@misc{wang2025targetbench,
  title={Target-Bench: Can World Models Achieve Mapless Path Planning with Semantic Targets?},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2511.17792}
}

• arXiv: 2511.17792