Multivariate TS Prediction Eval

multivariate-ts-prediction-eval

Spatiotemporal Attention for Multivariate Time Series Prediction and Interpretation — Gangopadhyay et al. (2020) (arXiv:2008.04882, 2020)

What this evaluates

Evaluates the ability of spatiotemporal attention models to accurately predict future values in multivariate time series across environmental, building HVAC, and clinical domains. It also probes the model's capacity to produce interpretable attention weights that align with known physical or physiological relationships.

Datasets

• Beijing PM2.5 Data Set — total 43792; splits: train (26275), val (8758), test (8759)
• Building HVAC Dataset — total 34887; splits: train (20932), val (6977), test (6978)
• MIMIC-III EHR Dataset — total 21139; splits: train (14681), val (3222), test (3236)

Metrics

• RMSE (primary) — range: other
  • Square root of the mean of squared differences between predicted and actual values.
• MAE — range: other
  • Mean of absolute differences between predicted and actual values.
• R² Score — range: [0, 1]
  • 1 minus the ratio of the sum of squared residuals to the total sum of squares. Measures the proportion of variance in the target explained by the model.

Input / output format

Input: Multivariate time series sequence of length $T_x$ (5 or 24 time steps) containing multiple sensor or clinical variables.

Output: Predicted continuous values for the target variable over the next $T_y$ time steps (3 or 4 steps).

Scoring recipe

def compute_metrics(y_true, y_pred):
    rmse = np.sqrt(np.mean((y_true - y_pred) ** 2))
    mae = np.mean(np.abs(y_true - y_pred))
    ss_res = np.sum((y_true - y_pred) ** 2)
    ss_tot = np.sum((y_true - np.mean(y_true)) ** 2)
    r2 = 1 - (ss_res / ss_tot)
    return {'RMSE': rmse, 'MAE': mae, 'R2': r2}

Common pitfalls

• The paper reports training time per epoch for neural baselines but total training time for SVR-RBF, making direct time comparisons misleading.
• Input sequence length ($T_x$) and prediction horizon ($T_y$) differ across datasets (Pollution/Building: $T_x=5, T_y=3/4$; EHR: $T_x=24, T_y=4$).
• Interpretability is assessed qualitatively by comparing attention weights to domain literature, not via a quantitative metric.

Evidence (verbatim from paper)

We use three evaluation metrics: root mean square error (RMSE), mean absolute error (MAE), and coefficient of determination or R-squared score ($R^{2}$). Tables 2, 3 and 4 presents the empirical results for the pollution, building and EHR datasets respectively.

Citation

@misc{gangopadhyay2020spatiotemporal,
  title={Spatiotemporal Attention for Multivariate Time Series Prediction and Interpretation},
  author={Gangopadhyay et al. (2020)},
  year={2020},
  note={arXiv:2008.04882}
}

• arXiv: 2008.04882