Feature Selection

Feature Selection

When to Use

Feature selection is an optional refinement, not a step on the critical path. It is usually run once a forecaster is already trained and tuned, to trim the feature set and reduce model complexity, rather than to improve accuracy. It is also the slowest step in the pipeline, since the selector refits the estimator many times.

Use feature selection when:

• You have many lags or exogenous variables and want to reduce overfitting
• You want to identify which features matter most
• You need to speed up inference or training by removing irrelevant features

After a large reduction, re-run hyperparameter-optimization: the best parameters for the full feature set are not necessarily best for the trimmed one.

Related skills

• Prerequisite: autocorrelation-and-lag-selection (generate an informed candidate set of lags before running the selector)
• Prerequisite: feature-engineering (create the rolling, calendar, and exogenous features that the selector will rank)
• Next: hyperparameter-optimization (re-tune the estimator on the reduced feature set)

Single Series

select_features works with ForecasterRecursive and ForecasterDirect.

from sklearn.feature_selection import RFECV
from sklearn.ensemble import RandomForestRegressor
from skforecast.recursive import ForecasterRecursive
from skforecast.preprocessing import RollingFeatures
from skforecast.feature_selection import select_features

# Create forecaster with many candidate features
rolling = RollingFeatures(stats=['mean', 'std', 'min', 'max'], window_sizes=[7, 14])

forecaster = ForecasterRecursive(
    estimator=RandomForestRegressor(n_estimators=100, random_state=123),
    lags=48,  # Many lags — feature selection will reduce
    window_features=rolling,
)

# Run feature selection
selected_lags, selected_window_features, selected_exog = select_features(
    forecaster=forecaster,
    selector=RFECV(
        estimator=RandomForestRegressor(n_estimators=50, random_state=123),
        step=1,
        cv=3,
    ),
    y=y_train,
    exog=exog_train,
    select_only=None,          # 'autoreg' (lags+window), 'exog', or None (all)
    force_inclusion=None,      # Features to always keep (list or regex string)
    subsample=0.5,             # Use 50% of data for faster selection
    random_state=123,
    verbose=True,
)

# Apply selected lags to the same forecaster (simplest approach)
forecaster.set_lags(selected_lags)

# selected_window_features is a list of names (strings), not the RollingFeatures
# object. Use the names to verify which window features were selected.
print(f'Selected window features: {selected_window_features}')
print(f'Selected exog variables: {selected_exog}')

Multi-Series

select_features_multiseries works with ForecasterRecursiveMultiSeries and ForecasterDirectMultiVariate.

Note: When used with ForecasterDirectMultiVariate, selected_lags is returned as a dict (one entry per series) instead of a list.

from skforecast.recursive import ForecasterRecursiveMultiSeries
from skforecast.feature_selection import select_features_multiseries

forecaster = ForecasterRecursiveMultiSeries(
    estimator=RandomForestRegressor(n_estimators=100, random_state=123),
    lags=48,
    encoding='ordinal',
)

selected_lags, selected_window_features, selected_exog = select_features_multiseries(
    forecaster=forecaster,
    selector=RFECV(
        estimator=RandomForestRegressor(n_estimators=50, random_state=123),
        step=1,
        cv=3,
    ),
    series=series_df,
    exog=exog_df,
    select_only=None,
    force_inclusion=None,
    subsample=0.5,
    random_state=123,
    verbose=True,
)

Force Inclusion

# Always keep specific features regardless of selection
selected_lags, selected_wf, selected_exog = select_features(
    forecaster=forecaster,
    selector=selector,
    y=y_train,
    exog=exog_train,
    force_inclusion=['temperature', 'holiday'],  # Always keep these exog columns
)

# Regex pattern to force include
selected_lags, selected_wf, selected_exog = select_features(
    forecaster=forecaster,
    selector=selector,
    y=y_train,
    exog=exog_train,
    force_inclusion='^lag_',  # Keep all lag features
)

Select Only Specific Feature Types

# Only select among exogenous variables (keep all lags)
selected_lags, selected_wf, selected_exog = select_features(
    forecaster=forecaster,
    selector=selector,
    y=y_train,
    exog=exog_train,
    select_only='exog',  # Only select exog, keep all autoregressive features
)

# Only select among autoregressive features (keep all exog)
selected_lags, selected_wf, selected_exog = select_features(
    forecaster=forecaster,
    selector=selector,
    y=y_train,
    exog=exog_train,
    select_only='autoreg',  # Only select lags+window features, keep all exog
)

Common Mistakes

1. Using the wrong selector: RFECV works best for recursive feature elimination. For faster selection, use SelectFromModel.
2. Too small subsample: If subsample is too small, selection may be unreliable. Use at least 0.3.
3. Not updating forecaster: After selection, update the forecaster with forecaster.set_lags(selected_lags) — the original is not modified in place by select_features.
4. Running on full dataset: Always run on training data only (y_train, exog_train).
5. Confusing selected_window_features with RollingFeatures: The returned selected_window_features is a list of feature name strings (e.g. ['mean_7', 'std_14']), not the RollingFeatures object itself. Use these names to verify which window features were kept, but pass the original RollingFeatures instance to the forecaster.