Robotracer Spatial Eval

robotracer-spatial-eval

RoboTracer: Mastering Spatial Trace with Reasoning in Vision-Language Models for Robotics — Enshen Zhou et al. (2025) (arXiv:2512.13660, 2025)

What this evaluates

Evaluates vision-language models' ability to perform spatial understanding, metric measuring, 2D/3D referring, and multi-step visual tracing in cluttered environments. It probes geometric reasoning, depth estimation, and collision-free path planning for robotic manipulation.

Datasets

• CV-Bench — total ?; splits: test (-1)
• BLINK_val — total ?; splits: val (-1)
• RoboSpatial — total ?; splits: test (-1)
• Embspacial — total ?; splits: test (-1)
• Q-spatial — total ?; splits: test (-1)
• MSMU — total ?; splits: test (-1)
• Where2Place — total ?; splits: test (-1)
• RefSpatial-Bench — total ?; splits: test (-1)
• ShareRobot-Bench — total ?; splits: test (-1)
• VABench-V — total ?; splits: test (-1)
• TraceSpatial-Bench — total 100; splits: test (100)
• RoboTwin — total 19; splits: test (19)
• MMEtest — total ?; splits: test (-1)
• MMBenchdev — total ?; splits: dev (-1)
• OK-VQA — total ?; splits: test (-1)
• POPE — total ?; splits: test (-1)

Metrics

• Top-1 success rate (%) (primary) — range: percent
  • Percentage of instances where the predicted spatial trace or referring point exactly matches the ground truth within a defined tolerance, with top-1 indicating the single best prediction.
• Discrete Fréchet Distance (DFD) — range: other
  • Measures the similarity between two trajectory curves (predicted vs. ground truth) in a discrete space. Lower values indicate better alignment.
• Hausdorff Distance (HD) — range: other
  • The maximum distance from a point in one set to the closest point in the other set. Lower values indicate tighter trace alignment.
• Root Mean Square Error (RMSE) — range: other
  • Square root of the average of squared differences between predicted and ground truth coordinates. Lower values indicate higher precision.

Input / output format

Input: RGB images, optionally augmented with camera intrinsics and absolute depth maps, paired with natural language instructions describing the spatial task.

Output: Predicted 2D masks/points, 3D bounding boxes or coordinates, and trajectory paths; evaluated as success/failure flags or distance metrics.

Scoring recipe

def evaluate(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if is_within_tolerance(p, g))
    success_rate = (correct / len(predictions)) * 100
    dfd = discrete_frechet_distance(predictions, gold)
    hd = hausdorff_distance(predictions, gold)
    rmse = np.sqrt(np.mean((np.array(predictions) - np.array(gold))**2))
    multi_step_success = all(is_collision_free(p) and is_correct_start_end(p, g) for p, g in zip(predictions, gold))
    return success_rate, dfd, hd, rmse, multi_step_success

Common pitfalls

• Models often produce 'floating' or colliding traces in 3D due to inaccurate depth estimation when only RGB inputs are provided.
• Evaluating 2D referring tasks without accounting for the decoupled point formulation or dimensionality reduction used in training can lead to unfair comparisons.
• Assuming success in 2D spatial referring implies success in 3D multi-step tracing, which requires explicit geometric reasoning and collision-free path validation.

Evidence (verbatim from paper)

Top-1/-2 success rate (%) are indicated by bold/underlined text. DFD/HD/RMSE are Discrete Fréchet Distance, Hausdorff Distance, Root Mean Square Error.

Citation

@misc{zhou2025robotracer,
  title={RoboTracer: Mastering Spatial Trace with Reasoning in Vision-Language Models for Robotics},
  author={Enshen Zhou et al. (2025)},
  year={2025},
  note={arXiv:2512.13660}
}

• arXiv: 2512.13660