qhjqhj00 maptrace-eval
MapTrace: Scalable Data Generation for Route Tracing on Maps — Panagopoulou et al. (2025) (arXiv:2512.19609, 2025)
What this evaluates
Evaluates fine-grained spatial reasoning and pixel-accurate route tracing on commercial map images. Models must generate precise path coordinates or masks corresponding to text-based navigation queries.
Datasets
- MapTrace — total 23000; splits: train (20000), val (3000)
- Map-Bench — total 1573; splits: test (1573)
Metrics
NDTW(primary) — range: [0, 1]- Normalized Dynamic Time Warping distance between the predicted path coordinates and the ground-truth path. Lower values indicate better alignment and accuracy.
Input / output format
Input: A map image paired with a text query specifying start/end points or route instructions.
Output: A sequence of pixel coordinates or a binary mask representing the traced path.
Scoring recipe
def compute_ndtw(pred_path, gt_path):
# pred_path and gt_path are lists of (x, y) coordinates
dist = dtw_distance(pred_path, gt_path)
norm_dist = dist / max(len(pred_path), len(gt_path))
return norm_dist
Common pitfalls
- Pixel-level precision is required; models often output topologically correct but spatially misaligned paths, which heavily penalizes NDTW scores.
- Evaluation queries on Map-Bench rely on human annotations that may vary in granularity, making cross-query comparison noisy.
- Baseline methods like Mask-Method extract paths from color masks, which can fail on maps with low contrast or overlapping features, skewing comparative results.
Evidence (verbatim from paper)
Fine-tuning MLLMs on this dataset improves path-tracing robustness (up to +6.4 points on MapBench) and reduces NDTW error, demonstrating that explicit supervision in coordinate-level spatial reasoning can bridge a critical gap in current models.
Citation
@misc{panagopoulou2025maptrace,
title={MapTrace: Scalable Data Generation for Route Tracing on Maps},
author={Panagopoulou et al. (2025)},
year={2025},
note={arXiv:2512.19609}
}
- arXiv: 2512.19609