Tardis Stride Eval

tardis-stride-eval

TARDIS STRIDE: A Spatio-Temporal Road Image Dataset and World Model for Autonomy — Carrión et al. (2025) (arXiv:2506.11302, 2025)

What this evaluates

Evaluates a generative world model's ability to produce controllable road images, predict geographic coordinates from street-view imagery, and generate self-consistent navigation actions on held-out spatiotemporal data.

Datasets

• STRIDE — total 3600000; splits: test (-1)

Metrics

• georeferencing_error_m (primary) — range: meters
  • Euclidean or great-circle distance in meters between predicted coordinates and ground truth. Reported as a CDF of prediction errors.
• road_adherence_rate — range: percent
  • Percentage of generated navigation actions that result in a state within a specified distance threshold (e.g., 4m) of the nearest road centerline.
• perplexity — range: other
  • Standard cross-entropy loss exponentiated over generated tokens/images as a function of spatial and temporal commands.

Input / output format

Input: Spatiotemporally held-out testing data consisting of panoramic road images, ground-truth states, and spatio-temporal commands (move distance/heading or temporal shifts).

Output: Generated images, predicted coordinate tokens (latitude/longitude), or generated navigation instructions (distance and heading).

Scoring recipe

def georeferencing_error(pred_latlon, true_latlon):
    return haversine_distance_m(pred_latlon, true_latlon)

def road_adherence_rate(actions, road_centerlines, threshold=4.0):
    valid = sum(1 for a in actions if distance_to_road(apply_action(a), road_centerlines) <= threshold)
    return (valid / len(actions)) * 100

def perplexity_metric(tokens):
    return exp(cross_entropy_loss(tokens))

Common pitfalls

• Testing data is strictly the bottom 10% of the training area, held out both spatially and temporally.
• Georeferencing baseline (SVG) is artificially restricted to a 50m search space to match TARDIS's max move action, which may not reflect real-world open-set performance.
• Road adherence threshold varies (1m-10m); the paper reports 77.4% using a 4m threshold based on average US lane length, not a fixed standard.
• Perplexity is used as a proxy for image quality, which the authors explicitly note is often not representative of actual visual fidelity.

Evidence (verbatim from paper)

We measure all metrics and visualize all results using purely testing data which has been kept completely outside the models training regime spatiotemporally (bottom 10 percent of the training area). We measure quality quantitatively via the perplexity metric. We prompt TARDIS with 1…n testing nodes and measure its real-world meter error compared to the real coordinate. We measure this quantitatively by calculating the percent of actions which produce a following state within some distance threshold of the nearest road center line.

Citation

@misc{carriion2025tardis,
  title={TARDIS STRIDE: A Spatio-Temporal Road Image Dataset and World Model for Autonomy},
  author={Carrión et al. (2025)},
  year={2025},
  note={arXiv:2506.11302}
}

• arXiv: 2506.11302