Backtesting Configuration
When to Use
Use this skill to translate a deployment scenario (retraining cadence, forecast
horizon, data budget, ingestion delay) into TimeSeriesFold parameters.
TimeSeriesFold is the cross-validation strategy passed via the cv argument to
backtesting_forecaster (and its multi-series / stats variants) and to the
hyperparameter search functions.
from skforecast.model_selection import backtesting_forecaster, TimeSeriesFold
cv = TimeSeriesFold(
steps=7, # forecast horizon
initial_train_size=365, # first training window (required)
refit=False, # train once (default)
fixed_train_size=True, # rolling window (default)
gap=0,
)
metric, predictions = backtesting_forecaster(
forecaster=forecaster,
y=data['target'],
cv=cv,
metric='mean_absolute_error',
)
For fast hyperparameter tuning where multi-step realism is not required, use
OneStepAheadFold instead: it validates one step ahead (no recursive
prediction), so it is much faster but less representative of multi-step
performance. Use TimeSeriesFold for realistic multi-step backtesting.
Related skills
- Prerequisite:
forecasting-single-series / forecasting-multiple-series (have a fitted forecaster before backtesting)
- Alongside:
metric-selection (choose the metric(s) backtesting reports)
- Alongside:
hyperparameter-optimization (the same cv object drives the search functions)
- Next:
prediction-intervals (add interval= / interval_method= to backtest uncertainty)
Stop Conditions
Scan before writing code. Each row lists a rule, the symptom when it is broken, and the recovery. Full pitfall catalog: the troubleshooting-common-errors skill.
| Rule |
Symptom |
Recovery |
initial_train_size must be provided (the API default is None, which only works when reusing an already-fitted forecaster) |
ValueError / no training in the first fold |
Pass an int, date string, or Timestamp, e.g. initial_train_size=len(y) - 100 |
initial_train_size does not accept a float fraction |
ValueError: must be int, date string, Timestamp, or None |
Convert fractions to an int, e.g. int(len(data) * 0.7) |
| Configuration must yield at least 2 folds |
Single-fold or empty backtest |
Ensure initial_train_size + gap + 2 * steps <= n_observations |
refit=True retrains every fold (slow path); the default is refit=False |
Backtest much slower than expected |
Use refit=False or an int cadence (e.g. refit=7) unless per-fold retraining is required |
gap, steps, and fold_stride count observations, not calendar time |
Off-by-frequency delays |
Convert the delay to the series frequency (e.g. 2 days at daily freq = gap=2) |
TimeSeriesFold Parameters
| Parameter |
Type |
Default |
Description |
initial_train_size |
int, str, pd.Timestamp |
None (required) |
Observations for initial training. Int = count, str/Timestamp = last training date. A common starting point is int(len(data) * 0.7). |
refit |
bool, int |
False |
Refit every fold (True), train once (False, default), or every n folds (int). |
fixed_train_size |
bool |
True |
Rolling fixed window (True, default) vs expanding window (False). |
gap |
int |
0 |
Observations between training end and test start. |
fold_stride |
int, None |
None (= steps) |
Distance between consecutive test set starts. |
skip_folds |
int, list, None |
None |
Skip folds to reduce compute. Int = keep every n-th. |
allow_incomplete_fold |
bool |
True |
Allow final fold with fewer observations than steps. |
Constraints
- Must produce at least 2 folds:
initial_train_size + gap + 2 * steps <= n_observations
initial_train_size must be large enough for the model to learn patterns (at minimum 2× the lag order for ML models, or 2× steps for statistical models)gap simulates real-world delay between data availability and forecast usage- When
fixed_train_size=True, the window rolls forward (oldest data discarded). Use for concept drift or when old data is less relevant.
- When
fixed_train_size=False, the window expands (all history retained). Use when more data always helps.
Business Scenario Mapping
Retraining frequency
| Scenario |
Configuration |
| Retrain every time new data arrives |
refit=True |
| Retrain weekly (with daily forecasts, steps=1) |
refit=7 |
| Retrain monthly (with daily forecasts, steps=7) |
refit=4 (every 4 folds × 7 steps ≈ monthly) |
| Never retrain / train once, evaluate across time |
refit=False (fixed_train_size has no effect without refit) |
Data freshness vs volume
| Scenario |
Configuration |
| Recent data more relevant (concept drift) |
fixed_train_size=True |
| All historical data valuable |
fixed_train_size=False (expanding) |
| Limited compute budget |
refit=False or skip_folds=3 (keep every 3rd fold) |
Deployment gap
| Scenario |
Configuration |
| Real-time predictions (no delay) |
gap=0 |
| 1-day delay between data collection and forecast |
gap=1 (if freq=daily) |
| Forecast must be ready before weekend (2-day gap) |
gap=2 |
Initial training size
| Scenario |
Configuration |
| Default (balanced) |
int(len(data) * 0.7) |
| Maximize evaluation coverage |
Minimum viable: 2 * max_lag or 2 * steps |
| Maximize training data |
n_observations - gap - 2 * steps (minimum 2 folds) |
| Start from specific date |
initial_train_size="2023-01-01" |
| Conservative (large training set) |
int(len(data) * 0.8) |
Fold stride (test set overlap)
| Scenario |
Configuration |
| Non-overlapping evaluation (default) |
fold_stride=None (equals steps) |
| Sliding window evaluation (1-step shift) |
fold_stride=1 |
| Sparse evaluation (save compute) |
fold_stride=steps * 2 |
Examples
"I retrain my model every Monday and forecast the next 7 days"
refit = True
fixed_train_size = False # Keep all history
gap = 0 # No delay
fold_stride = None # Non-overlapping weeks
"I want to simulate deploying once and seeing how the model degrades"
refit = False # Train once; fixed_train_size has no effect without refit
gap = 0
"There's a 2-day lag between data ingestion and when forecasts are needed"
gap = 2
refit = True
"I have limited compute — evaluate every other week"
refit = True
skip_folds = 2 # Keep every 2nd fold
"I want maximum evaluation coverage with a 12-step horizon"
initial_train_size = <minimum viable> # 2 * max_lag
refit = False # Faster; trains once, fixed_train_size has no effect
fold_stride = 1 # Sliding window (many overlapping folds)
1---2name: backtesting-configuration3description: Configures TimeSeriesFold parameters for backtesting based on deployment scenarios. Maps business requirements (retraining frequency, forecast horizon, data budget) to cross-validation strategy parameters. Use when the user describes how they plan to deploy or evaluate a model.4---56# Backtesting Configuration78## When to Use910Use this skill to translate a deployment scenario (retraining cadence, forecast11horizon, data budget, ingestion delay) into `TimeSeriesFold` parameters.12`TimeSeriesFold` is the cross-validation strategy passed via the `cv` argument to13`backtesting_forecaster` (and its multi-series / stats variants) and to the14hyperparameter search functions.1516```python17from skforecast.model_selection import backtesting_forecaster, TimeSeriesFold1819cv = TimeSeriesFold(20 steps=7, # forecast horizon21 initial_train_size=365, # first training window (required)22 refit=False, # train once (default)23 fixed_train_size=True, # rolling window (default)24 gap=0,25)2627metric, predictions = backtesting_forecaster(28 forecaster=forecaster,29 y=data['target'],30 cv=cv,31 metric='mean_absolute_error',32)33```3435For fast hyperparameter tuning where multi-step realism is not required, use36`OneStepAheadFold` instead: it validates one step ahead (no recursive37prediction), so it is much faster but less representative of multi-step38performance. Use `TimeSeriesFold` for realistic multi-step backtesting.3940### Related skills4142- **Prerequisite**: `forecasting-single-series` / `forecasting-multiple-series` (have a fitted forecaster before backtesting)43- **Alongside**: `metric-selection` (choose the metric(s) backtesting reports)44- **Alongside**: `hyperparameter-optimization` (the same `cv` object drives the search functions)45- **Next**: `prediction-intervals` (add `interval=` / `interval_method=` to backtest uncertainty)4647## Stop Conditions4849Scan before writing code. Each row lists a rule, the symptom when it is broken, and the recovery. Full pitfall catalog: the `troubleshooting-common-errors` skill.5051| Rule | Symptom | Recovery |52|------|---------|----------|53| `initial_train_size` must be provided (the API default is `None`, which only works when reusing an already-fitted forecaster) | `ValueError` / no training in the first fold | Pass an int, date string, or Timestamp, e.g. `initial_train_size=len(y) - 100` |54| `initial_train_size` does not accept a float fraction | `ValueError`: must be int, date string, Timestamp, or None | Convert fractions to an int, e.g. `int(len(data) * 0.7)` |55| Configuration must yield at least 2 folds | Single-fold or empty backtest | Ensure `initial_train_size + gap + 2 * steps <= n_observations` |56| `refit=True` retrains every fold (slow path); the default is `refit=False` | Backtest much slower than expected | Use `refit=False` or an int cadence (e.g. `refit=7`) unless per-fold retraining is required |57| `gap`, `steps`, and `fold_stride` count observations, not calendar time | Off-by-frequency delays | Convert the delay to the series frequency (e.g. 2 days at daily freq = `gap=2`) |5859## TimeSeriesFold Parameters6061| Parameter | Type | Default | Description |62|-----------|------|---------|-------------|63| `initial_train_size` | int, str, pd.Timestamp | None (required) | Observations for initial training. Int = count, str/Timestamp = last training date. A common starting point is `int(len(data) * 0.7)`. |64| `refit` | bool, int | False | Refit every fold (True), train once (False, default), or every n folds (int). |65| `fixed_train_size` | bool | True | Rolling fixed window (True, default) vs expanding window (False). |66| `gap` | int | 0 | Observations between training end and test start. |67| `fold_stride` | int, None | None (= steps) | Distance between consecutive test set starts. |68| `skip_folds` | int, list, None | None | Skip folds to reduce compute. Int = keep every n-th. |69| `allow_incomplete_fold` | bool | True | Allow final fold with fewer observations than steps. |7071## Constraints7273- Must produce **at least 2 folds**: `initial_train_size + gap + 2 * steps <= n_observations`74- `initial_train_size` must be large enough for the model to learn patterns (at minimum 2× the lag order for ML models, or 2× steps for statistical models)75- `gap` simulates real-world delay between data availability and forecast usage76- When `fixed_train_size=True`, the window rolls forward (oldest data discarded). Use for concept drift or when old data is less relevant.77- When `fixed_train_size=False`, the window expands (all history retained). Use when more data always helps.7879## Business Scenario Mapping8081### Retraining frequency8283| Scenario | Configuration |84|----------|--------------|85| Retrain every time new data arrives | `refit=True` |86| Retrain weekly (with daily forecasts, steps=1) | `refit=7` |87| Retrain monthly (with daily forecasts, steps=7) | `refit=4` (every 4 folds × 7 steps ≈ monthly) |88| Never retrain / train once, evaluate across time | `refit=False` (`fixed_train_size` has no effect without refit) |8990### Data freshness vs volume9192| Scenario | Configuration |93|----------|--------------|94| Recent data more relevant (concept drift) | `fixed_train_size=True` |95| All historical data valuable | `fixed_train_size=False` (expanding) |96| Limited compute budget | `refit=False` or `skip_folds=3` (keep every 3rd fold) |9798### Deployment gap99100| Scenario | Configuration |101|----------|--------------|102| Real-time predictions (no delay) | `gap=0` |103| 1-day delay between data collection and forecast | `gap=1` (if freq=daily) |104| Forecast must be ready before weekend (2-day gap) | `gap=2` |105106### Initial training size107108| Scenario | Configuration |109|----------|--------------|110| Default (balanced) | `int(len(data) * 0.7)` |111| Maximize evaluation coverage | Minimum viable: `2 * max_lag` or `2 * steps` |112| Maximize training data | `n_observations - gap - 2 * steps` (minimum 2 folds) |113| Start from specific date | `initial_train_size="2023-01-01"` |114| Conservative (large training set) | `int(len(data) * 0.8)` |115116### Fold stride (test set overlap)117118| Scenario | Configuration |119|----------|--------------|120| Non-overlapping evaluation (default) | `fold_stride=None` (equals steps) |121| Sliding window evaluation (1-step shift) | `fold_stride=1` |122| Sparse evaluation (save compute) | `fold_stride=steps * 2` |123124## Examples125126### "I retrain my model every Monday and forecast the next 7 days"127```128refit = True129fixed_train_size = False # Keep all history130gap = 0 # No delay131fold_stride = None # Non-overlapping weeks132```133134### "I want to simulate deploying once and seeing how the model degrades"135```136refit = False # Train once; fixed_train_size has no effect without refit137gap = 0138```139140### "There's a 2-day lag between data ingestion and when forecasts are needed"141```142gap = 2143refit = True144```145146### "I have limited compute — evaluate every other week"147```148refit = True149skip_folds = 2 # Keep every 2nd fold150```151152### "I want maximum evaluation coverage with a 12-step horizon"153```154initial_train_size = <minimum viable> # 2 * max_lag155refit = False # Faster; trains once, fixed_train_size has no effect156fold_stride = 1 # Sliding window (many overlapping folds)157```
skforecast