Dejavu Forecasting Eval

dejavu-forecasting-eval

D'ej`a vu: A data-centric forecasting approach through time series cross-similarity — Kang et al. (2019) (arXiv:1909.00221, 2019)

What this evaluates

Evaluates time series forecasting accuracy and prediction interval calibration using a data-centric cross-similarity approach. It probes the model's ability to aggregate future paths from similar historical reference series to generate point forecasts and uncertainty bounds across different frequencies and historical sample lengths.

Datasets

• M1 and M3 forecasting competitions — total 3830; splits: test (3830)

Metrics

• MASE (primary) — range: other
  • Mean Absolute Scaled Error. Scaled version of MAE where the scaling factor is the mean absolute error of the seasonal naive forecast on historical data. Lower values indicate better accuracy.
• Coverage — range: percent
  • Proportion of true values falling within the predicted lower and upper interval bounds over the forecasting horizon. Target is 95%.
• Upper coverage — range: percent
  • Proportion of true values not exceeding the upper interval bound. Serves as a proxy for achieved service levels. Target is 97.5%.
• Spread — range: other
  • Mean difference between upper and lower bounds, scaled by the seasonal naive error on historical data. Serves as a proxy for holding costs. Lower is better.
• MSIS — range: other
  • Mean Scaled Interval Score. Calculated as in Equation 3 of the paper. Lower values indicate better interval accuracy and sharpness.

Input / output format

Input: Historical time series observations (optionally truncated to specific lengths), data frequency (yearly, quarterly, monthly), and seasonal period length s.

Output: Point forecasts f_t for horizon h, and lower/upper prediction interval bounds L_t, U_t for alpha=0.05.

Scoring recipe

def compute_mase(y_true, y_pred, y_hist, s, h, n):
    mae = sum(abs(y_true[t] - y_pred[t]) for t in range(n+1, n+h+1)) / h
    naive_err = sum(abs(y_hist[t] - y_hist[t-s]) for t in range(s+1, n+1)) / (n-s)
    return mae / naive_err

def compute_coverage(y_true, L, U, h, n):
    return sum(1 for t in range(n+1, n+h+1) if L[t] < y_true[t] < U[t]) / h

Common pitfalls

• Averaging MASE across series is statistically valid due to its scale-independence, unlike MAPE which can be heavily skewed by near-zero actuals.
• Target coverage is 95%, but upper coverage target is 97.5% due to its one-sided nature; deviation from these targets indicates under- or over-coverage.
• Preprocessing (seasonal adjustment and Loess smoothing) is critical; skipping it significantly degrades accuracy, with monthly data requiring more smoothing than yearly/quarterly data.

Evidence (verbatim from paper)

The point forecast accuracy is measured in terms of the Mean Absolute Scaled Error (MASE: Hyndman & Koehler, 2006). MASE is a scaled version of the mean absolute error, with the scaling being the mean absolute error of the seasonal naive for the historical data. ... Across all horizons of a single series, the MASE value can be calculated as MASE = (1/h) * sum(|y_t - f_t|) / ((1/(n-s)) * sum(|y_t - y_{t-s}|), where y_t and f_t are the actual observation and the forecast for period t, n is the sample size, s is the length of the seasonal period, and h is the forecasting horizon. Lower MASE values are better.

Citation

@misc{kang2019dejavu,
  title={D'ej\`a vu: A data-centric forecasting approach through time series cross-similarity},
  author={Kang et al. (2019)},
  year={2019},
  note={arXiv:1909.00221}
}

• arXiv: 1909.00221