Mobility Timeseries Eval

mobility-timeseries-eval

Timeseries Foundation Models for Mobility: A Benchmark Comparison with Traditional and Deep Learning Models — Graser (2025) (arXiv:2504.03725, 2025)

What this evaluates

Evaluates the accuracy of time series forecasting models on urban mobility data across different prediction horizons. It probes how well traditional, deep learning, and foundation models capture short-term, medium-term, and long-term temporal dependencies in bike-sharing flows.

Datasets

• BikeNYC — total ?; splits: test (-1)
• BikeVIE — total ?; splits: test (-1)

Metrics

• RMSE (primary) — range: other
  • Root Mean Square Error, calculated as the square root of the mean of squared differences between predicted and actual values across the forecast horizon.

Input / output format

Input: Hourly time series of bike-sharing counts (flows or station availability) for each grid cell or station.

Output: Forecasted bike-sharing counts for 1-hour, 12-hour, and 24-hour prediction horizons.

Scoring recipe

def compute_rmse(predictions, actuals):
    n = len(predictions)
    squared_errors = [(p - a) ** 2 for p, a in zip(predictions, actuals)]
    return (sum(squared_errors) / n) ** 0.5

Common pitfalls

• Uses a rolling window backtesting approach on only the last 10 days, which may not represent full seasonal cycles.
• BikeVIE dataset has a hard cutoff at mid-August 2019 to avoid seasonal gaps, limiting long-term evaluation.
• Forecast horizons (1h, 12h, 24h) are evaluated separately, so models must be compared at each horizon independently.

Evidence (verbatim from paper)

Performance is evaluated using Root Mean Square Error (RMSE) for 1-hour, 12-hour, and 24-hour forecasts, with the last ten days used for backtesting via a rolling window approach.

Citation

@misc{graser2025timeseries,
  title={Timeseries Foundation Models for Mobility: A Benchmark Comparison with Traditional and Deep Learning Models},
  author={Graser (2025)},
  year={2025},
  note={arXiv:2504.03725}
}

• arXiv: 2504.03725