jeremylongshore Cross-Validator Skill
Evaluates time series model performance using cross-validation.
Purpose
Rigorously assesses how well a time series model generalizes to unseen data by simulating future predictions.
Overview
This skill automates time series cross-validation by splitting historical data into multiple training and validation sets based on expanding or sliding window techniques. It integrates with TimeGPT and StatsForecast to evaluate model performance across various time periods. It reports key accuracy metrics, helping users select the best model.
Prerequisites
Tools: Read, Write, Bash, Glob, Grep
Environment: NIXTLA_TIMEGPT_API_KEY (if using TimeGPT)
Packages:
pip install nixtla pandas statsforecast matplotlib
Instructions
Step 1: Prepare data
Read time series data from CSV file into a pandas DataFrame using the data loader script.
Script: {baseDir}/scripts/load_data.py
The script expects a CSV file with columns: unique_id, ds (timestamp), and y (target value).
Example usage:
python {baseDir}/scripts/load_data.py data.csv
Step 2: Configure cross-validation
Define parameters like window size, step size, and number of folds using the configuration script.
Script: {baseDir}/scripts/configure_cv.py
The script creates expanding window splits for cross-validation. It validates that the data is sufficient for the specified window size and number of folds.
Step 3: Execute cross-validation
Run the cross-validation script with your chosen model and parameters.
Script: {baseDir}/scripts/cross_validate.py
Usage:
python {baseDir}/scripts/cross_validate.py \
--input data.csv \
--model arima \
--window 20 \
--folds 3 \
--freq D
Supported models:
timegpt: TimeGPT API (requires NIXTLA_TIMEGPT_API_KEY)arima: AutoARIMA from StatsForecastets: AutoETS from StatsForecasttheta: AutoTheta from StatsForecastnaive: SeasonalNaive baseline
Step 4: Analyze results
The script automatically calculates and outputs cross-validation metrics (MAE, RMSE) for all folds.
Output
- cv_results.csv: CSV file containing the cross-validation results for each fold.
- metrics.json: JSON file containing overall performance metrics across all folds.
- plots/: Directory containing plots comparing actual vs. predicted values for each fold.
Error Handling
Error:
Input file not foundSolution: Ensure the specified input CSV file exists at the given path.Error:
Invalid model nameSolution: Use a supported model name: 'timegpt', 'arima', 'ets', 'theta', 'naive'.Error:
Insufficient data for cross-validationSolution: Increase the length of the input time series or reduce the window size.Error:
Missing required parameterSolution: Specify all required parameters: input, model, window, folds.Error:
NIXTLA_TIMEGPT_API_KEY environment variable not set.Solution: Set theNIXTLA_TIMEGPT_API_KEYenvironment variable before running the script when using TimeGPT.
Examples
Example 1: Cross-validating TimeGPT on daily sales
Input:
unique_id,ds,y
store_1,2023-01-01,10
store_1,2023-01-02,12
store_1,2023-01-03,15
...
store_1,2023-12-31,20
Command:
python {baseDir}/scripts/cross_validate.py \
--input sales.csv \
--model timegpt \
--window 30 \
--folds 4 \
--freq D
Output:
fold,unique_id,ds,y,y_hat
1,store_1,2023-11-01,18,17.5
1,store_1,2023-11-02,20,19.2
...
Example 2: Cross-validating ARIMA on monthly demand
Input:
unique_id,ds,y
product_1,2020-01-01,100
product_1,2020-02-01,110
...
product_1,2023-12-01,125
Command:
python {baseDir}/scripts/cross_validate.py \
--input demand.csv \
--model arima \
--window 6 \
--folds 3 \
--freq M
Output:
{
"MAE": 5.2,
"RMSE": 7.1
}
Resources
- StatsForecast documentation: https://nixtlaverse.nixtla.io/statsforecast/
- TimeGPT API documentation: https://docs.nixtla.io/
- Cross-validation best practices: https://otexts.com/fpp3/tscv.html
- Scripts:
{baseDir}/scripts/directory contains all executable code