Lstnet Time Series Forecasting Eval

lstnet-time-series-forecasting-eval

Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks — Lai et al. (2017) (arXiv:1703.07015, 2017)

What this evaluates

Evaluates multivariate time series forecasting models on their ability to capture short-term local dependencies and long-term periodic patterns across diverse real-world datasets with varying temporal scales and frequencies.

Datasets

• Traffic — total 17544; splits: train (-1), val (-1), test (-1)
• Solar-Energy — total 52560; splits: train (-1), val (-1), test (-1)
• Electricity — total 26304; splits: train (-1), val (-1), test (-1)
• Exchange-Rate — total 7588; splits: train (-1), val (-1), test (-1)

Metrics

• RSE (primary) — range: [0, 1] or higher
  • Root Relative Squared Error: RSE = sqrt(Σ(Y_it - Ŷ_it)^2) / sqrt(Σ(Y_it - mean(Y))^2). It is a scale-insensitive version of RMSE normalized by a baseline mean prediction. Lower values indicate better performance.
• CORR — range: [-1, 1]
  • Empirical Correlation Coefficient: CORR = (1/n) Σ_i [ Σ_t((Y_it - mean(Y_i))(Ŷ_it - mean(Ŷ_i))) / sqrt(Σ_t(Y_it - mean(Y_i))^2 * Σ_t(Ŷ_it - mean(Ŷ_i))^2) ]. It measures the average Pearson correlation across all D variables. Higher values indicate better performance.

Input / output format

Input: Multivariate time series sequences of length T with D variables, sampled at fixed intervals (hourly, 10-min, or daily). Models receive historical windows and predict future values over horizons of 3, 6, 12, or 24 time steps depending on the dataset.

Output: Predicted values for D variables over the specified forecasting horizon.

Scoring recipe

def compute_rse(y_true, y_pred, y_mean):
    num = np.sqrt(np.sum((y_true - y_pred)**2))
    den = np.sqrt(np.sum((y_true - y_mean)**2))
    return num / den

def compute_corr(y_true, y_pred):
    n = y_true.shape[0]
    total = 0.0
    for i in range(n):
        yt, yp = y_true[i], y_pred[i]
        myt, myp = np.mean(yt), np.mean(yp)
        cov = np.sum((yt - myt) * (yp - myp))
        denom = np.sqrt(np.sum((yt - myt)**2) * np.sum((yp - myp)**2))
        total += cov / denom
    return total / n

Common pitfalls

• Forecasting horizons are dataset-specific (hours for Traffic/Electricity, minutes for Solar-Energy, days for Exchange-Rate), so direct cross-dataset metric comparison is invalid without horizon normalization.
• Single-output baselines (AR, LRidge, LSVR, GP) are trained independently per variable, whereas multivariate models share parameters; this architectural difference affects fairness in comparison.
• RSE is scale-insensitive and lower is better, whereas CORR is higher is better; confusing the optimization direction leads to incorrect model selection.

Evidence (verbatim from paper)

We used three conventional evaluation metrics defined as: Root Relative Squared Error (RSE): ... Empirical Correlation Coefficient (CORR) ... where Y, Ŷ ∈ R^{n×T} are ground true signals and system prediction signals, respectively. ... All datasets have been split into training set (60%), validation set (20%) and test set (20%) in chronological order.

Citation

@misc{lai2017lstnet,
  title={Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks},
  author={Lai et al. (2017)},
  year={2017},
  note={arXiv:1703.07015}
}

• arXiv: 1703.07015