Timerrecipe Eval

timerrecipe-eval

TimeRecipe: A Time-Series Forecasting Recipe via Benchmarking Module Level Effectiveness — Zhao et al. (2025) (arXiv:2506.06482, 2025)

What this evaluates

Evaluates the effectiveness of individual architectural modules (e.g., normalization, decomposition, embedding, feedforward types) across diverse time-series forecasting scenarios to identify optimal configurations and predict performance without training.

Datasets

• PEMS03 — total ?; splits: test (-1)
• ETT — total ?; splits: test (-1)
• Electricity — total ?; splits: test (-1)
• Social (Unemployment) — total ?; splits: test (-1)

Metrics

• MSE (primary) — range: other
  • Mean Squared Error: average of squared differences between predicted and actual time-series values. Lower values indicate better forecasting accuracy.
• MAE — range: other
  • Mean Absolute Error: average of absolute differences between predicted and actual time-series values. Lower values indicate better forecasting accuracy.
• Rank — range: other
  • Position of a model configuration when sorted by forecasting error (lower MSE/MAE is better). Rank 1 denotes the best-performing configuration in a given scenario.

Input / output format

Input: Multivariate or univariate time-series sequences with specified forecasting horizon and lookback window, along with computed dataset characteristics (e.g., seasonality, trend, stationarity, correlation, n-feature, HL-Ratio).

Output: Forecasted time-series values for the target horizon, and a predicted rank score for model configuration selection.

Scoring recipe

def compute_mse(preds, targets):
    return np.mean((preds - targets) ** 2)

def compute_mae(preds, targets):
    return np.mean(np.abs(preds - targets))

def compute_rank(configs, error_dict):
    sorted_configs = sorted(configs, key=lambda c: error_dict[c])
    return {c: i+1 for i, c in enumerate(sorted_configs)}

Common pitfalls

• Assuming a single architecture consistently outperforms others; the benchmark shows optimal modules depend heavily on dataset properties like trend regularity or horizon length.
• Treating the training-free selection as a trained model; it uses a LightGBM regressor to predict ranks based on dataset characteristics without fine-tuning the forecasting models.
• Ignoring module interactions; e.g., instance normalization improves performance on most datasets but degrades it on PEMS datasets.

Evidence (verbatim from paper)

For example, in the short-term multivariate forecasting on the PEMS03 dataset with a horizon of 12, the top-ranked configuration achieves an MSE of 0.714, outperforming iTransformer, one of the best existing forecasting models also covered by TIMERECIPE, which attains an MSE of 0.739 and ranks only 7th among all evaluated design combinations.

Citation

@misc{zhao2025timerrecipe,
  title={TimeRecipe: A Time-Series Forecasting Recipe via Benchmarking Module Level Effectiveness},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2506.06482}
}

• arXiv: 2506.06482