Mobility Forecasting Eval

mobility-forecasting-eval

Differentially Private Multivariate Time Series Forecasting of Aggregated Human Mobility With Deep Learning: Input or Gradient Perturbation? — Héber H. Arcolezi et al. (arXiv:2205.00436, 2022)

What this evaluates

This benchmark evaluates the accuracy of deep learning models for multivariate time series forecasting of aggregated human mobility across six urban regions. It specifically probes how well models can predict short-term (30-minute) passenger counts while preserving differential privacy through input or gradient perturbation mechanisms.

Datasets

• Paris Mobility Dataset — total 3456; splits: train (3120), test (336)

Metrics

• RMSE (primary) — range: other
  • Root Mean Square Error, calculated as the square root of the mean squared difference between true and predicted values: RMSE = (1/n_t) * sqrt(sum((y_t - y_hat_t)^2)).
• MAE — range: other
  • Mean Absolute Error, calculated as the mean of the absolute differences between true and predicted values: MAE = (1/n_t) * sum(|y_t - y_hat_t|).

Input / output format

Input: 6 prior time steps (lag values) of people counts per coarse region, augmented with cyclical time-of-day and time-of-week features.

Output: Predicted people count for the next 30-minute interval for each of the 6 coarse regions.

Scoring recipe

def compute_rmse(y_true, y_pred):
    return np.sqrt(np.mean((y_true - y_pred) ** 2))

def compute_mae(y_true, y_pred):
    return np.mean(np.abs(y_true - y_pred))

# Report per region and averaged across 6 regions

Common pitfalls

• Metrics are reported in real scale (absolute counts), so higher RMSE/MAE values do not inherently indicate worse performance for regions with larger populations.
• The reported epsilon is per training sample, but the overall privacy budget uses sequential composition over all 3120 samples, meaning the actual per-user privacy guarantee is significantly weaker than the per-sample epsilon suggests.

Evidence (verbatim from paper)

Performance metrics. All models were evaluated with standard time-series metrics, namely, root mean square error (RMSE) calculated as $RMSE=\frac{1}{n_{t}}\sqrt{\sum_{t=1}^{n_{t}}\left(y_{t}-\hat{y}{t}\right)^{2}}$ and mean absolute error (MAE) calculated as $MAE=\frac{1}{n{t}}\sum_{t=1}^{n_{t}}|y_{t}-\hat{y}{t}|$; in which $y{t}$ is the real output, $\hat{y}{t}$ is the predicted output, and $n{t}$ is the total number of samples in the test set, for $t\in[1,n_{t}]$. RMSE was the primary metric to select the final DL models.

Citation

@misc{arcolezi2022differentially,
  title={Differentially Private Multivariate Time Series Forecasting of Aggregated Human Mobility With Deep Learning: Input or Gradient Perturbation?},
  author={Héber H. Arcolezi et al.},
  year={2022},
  note={arXiv:2205.00436}
}

• arXiv: 2205.00436