Metric Selection

Metric Selection

When to Use

Use this skill when the user needs help choosing an evaluation metric, comparing metrics, understanding trade-offs between error measures, or configuring the metric parameter in backtesting_forecaster, bayesian_search_forecaster, or any other model selection function.

Related skills

• Prerequisite: choosing-a-forecaster (the forecaster type determines which metrics apply)
• Alongside: baseline-forecasting (MASE < 1 means the model beats the naive baseline)
• Next: hyperparameter-optimization (the chosen metric drives the search objective)
• Next: prediction-intervals (probabilistic metrics evaluate interval quality)

Quick Recommendations

If unsure, start here:

Point forecast metrics (pass as metric= in backtesting/search):

Probabilistic metrics (computed post-hoc on interval/quantile predictions):

Step 1 — What Are You Evaluating?

What is your forecaster producing?
│
├─► Point forecasts (single value per step)
│   └─► Go to Step 2
│
├─► Prediction intervals (lower_bound, upper_bound)
│   └─► Go to Step 3
│
├─► Quantile predictions (multiple quantile levels)
│   └─► Go to Step 3
│
├─► Class labels (ForecasterRecursiveClassifier)
│   └─► Go to Step 4
│
└─► Multi-series with aggregation needs
    └─► Go to Step 5 (then return to Step 2 or 3 for the base metric)

Step 2 — Point Forecast Metrics

Decision Table

Metric Properties

Full property matrix (scale independence, robustness, zero handling, y_train requirement, range and interpretation) for every metric: references/metric-compatibility.md.

Using Metrics in Code

Direct computation (train/test split)

For a simple train/test evaluation, import the metric and call it directly. Sklearn metrics are imported from sklearn.metrics; skforecast-specific metrics (mean_absolute_scaled_error, root_mean_squared_scaled_error, symmetric_mean_absolute_percentage_error) from skforecast.metrics.

from sklearn.metrics import mean_squared_error
from skforecast.metrics import mean_absolute_scaled_error

forecaster.fit(y=data_train)
predictions = forecaster.predict(steps=36)

# Sklearn metrics: func(y_true, y_pred)
error_mse = mean_squared_error(y_true=data_test, y_pred=predictions)

# Skforecast metrics that need y_train: func(y_true, y_pred, y_train)
error_mase = mean_absolute_scaled_error(
    y_true=data_test, y_pred=predictions, y_train=data_train
)

Inside backtesting

When using backtesting or hyperparameter search, pass metrics as strings — no import needed, and y_train is handled automatically:

from skforecast.model_selection import backtesting_forecaster, TimeSeriesFold

cv = TimeSeriesFold(steps=12, initial_train_size=len(y_train), refit=False)

# Single metric (pass as string)
metric, predictions = backtesting_forecaster(
    forecaster=forecaster,
    y=data['target'],
    cv=cv,
    metric='mean_absolute_error',
)

# Multiple metrics at once
metric, predictions = backtesting_forecaster(
    forecaster=forecaster,
    y=data['target'],
    cv=cv,
    metric=['mean_absolute_error', 'mean_absolute_scaled_error'],
)
# metric is a DataFrame with one column per metric

MASE and RMSSE require training data to compute the naive forecast baseline. When passed as a string or callable, skforecast handles y_train automatically — it detects whether the function signature includes a y_train parameter and provides the training data if so.

# Both options work identically in backtesting — y_train is handled internally
metric = 'mean_absolute_scaled_error'            # Option 1: as string
metric = mean_absolute_scaled_error              # Option 2: as callable (import from skforecast.metrics)

Step 3 — Probabilistic Forecast Metrics

Use these metrics when evaluating prediction intervals or quantile forecasts.

Coverage

Coverage measures the proportion of true values that fall within the predicted interval. Target: match the nominal level (e.g., 90% interval should have ~90% coverage).

from skforecast.metrics import calculate_coverage

coverage = calculate_coverage(
    y_true=y_test,
    lower_bound=predictions['lower_bound'],
    upper_bound=predictions['upper_bound'],
)
# Ideal: coverage ≈ 0.90 for a 90% interval
# coverage >> 0.90 → intervals too wide (not sharp)
# coverage << 0.90 → intervals too narrow (miscalibrated)

CRPS (Continuous Ranked Probability Score)

CRPS is a proper scoring rule that rewards both calibration and sharpness. Lower is better.

from skforecast.metrics import crps_from_predictions, crps_from_quantiles
import numpy as np

# From bootstrap samples (e.g., predict_bootstrapping output)
crps = crps_from_predictions(
    y_true=100.0,
    y_pred=np.array([98.2, 101.5, 99.8, 102.1, 97.5])  # Bootstrap predictions
)

# From quantile predictions (e.g., foundation model output)
crps = crps_from_quantiles(
    y_true=100.0,
    pred_quantiles=np.array([90.0, 95.0, 100.5, 105.0, 110.0]),
    quantile_levels=np.array([0.1, 0.25, 0.5, 0.75, 0.9]),
)

Winkler Score (Interval Score)

Scores a single prediction interval: it rewards narrow intervals but adds a penalty, scaled by alpha, whenever the true value falls outside. Strictly proper scoring rule; lower is better. alpha must match the interval's nominal level (e.g. an 80% interval given by the 10th/90th percentiles uses alpha=0.2).

from skforecast.metrics import winkler_score

# 80% interval -> alpha = 0.2
score = winkler_score(
    y_true=y_test,
    lower_bound=predictions['lower_bound'],
    upper_bound=predictions['upper_bound'],
    alpha=0.2,
)

Weighted Interval Score (WIS)

Generalises the Winkler Score to K intervals plus the median forecast, producing a single number that behaves like a discrete approximation of CRPS. Ideal when a model outputs several quantile levels (foundation models, ForecasterStats, or bootstrapping). y_pred is the median (0.5 quantile); alphas lists the K interval levels and must line up, by position, with the columns of the bound arrays. Lower is better.

import numpy as np
from skforecast.metrics import weighted_interval_score

# Two intervals: 80% (alpha=0.2) and 95% (alpha=0.05)
score = weighted_interval_score(
    y_true=y_test.to_numpy(),
    y_pred=predictions_median,                   # 0.5 quantile
    lower_bounds=np.column_stack([q10, q025]),   # columns match `alphas` order
    upper_bounds=np.column_stack([q90, q975]),
    alphas=[0.2, 0.05],
)

Note: Like calculate_coverage and CRPS, winkler_score and weighted_interval_score are computed post-hoc. Their signatures differ from func(y_true, y_pred), so they cannot be passed as a metric= string in backtesting or hyperparameter search.

Pinball Loss (Quantile Loss)

Evaluates a single quantile forecast. Use create_mean_pinball_loss(alpha) to create a metric function for a specific quantile level.

from skforecast.metrics import create_mean_pinball_loss

# Create metric for the 90th percentile
pinball_90 = create_mean_pinball_loss(alpha=0.9)

# Use in backtesting (as callable)
metric, predictions = backtesting_forecaster(
    forecaster=forecaster,
    y=data['target'],
    cv=cv,
    metric=pinball_90,
)

Step 4 — Classification Metrics

For ForecasterRecursiveClassifier only. These metrics evaluate predicted class labels, not continuous values.

from skforecast.recursive import ForecasterRecursiveClassifier
from sklearn.ensemble import RandomForestClassifier
from skforecast.model_selection import backtesting_forecaster, TimeSeriesFold

forecaster = ForecasterRecursiveClassifier(
    estimator=RandomForestClassifier(n_estimators=100, random_state=123),
    lags=24,
)

# y contains class labels (e.g., 'low', 'medium', 'high'); encoding is handled internally
cv = TimeSeriesFold(steps=10, initial_train_size=len(y) - 100, refit=False)

metric, predictions = backtesting_forecaster(
    forecaster=forecaster,
    y=y,
    cv=cv,
    metric='balanced_accuracy_score',
)

Step 5 — Multi-Series Metric Aggregation

When predicting multiple series, skforecast computes per-series metrics and can aggregate them into a single score. The aggregation options are:

In backtesting

Use add_aggregated_metric=True (default) to include all three aggregations:

from skforecast.model_selection import backtesting_forecaster_multiseries

metric, predictions = backtesting_forecaster_multiseries(
    forecaster=forecaster_multi,
    series=series_df,
    cv=cv,
    metric='mean_absolute_error',
    add_aggregated_metric=True,  # Default: includes average, weighted_average, pooling
)
# metric DataFrame has rows for each level + aggregated rows

In hyperparameter search

Use the aggregate_metric parameter to select which aggregations to report:

from skforecast.model_selection import bayesian_search_forecaster_multiseries

results, study = bayesian_search_forecaster_multiseries(
    forecaster=forecaster_multi,
    series=series_df,
    cv=cv,
    search_space=search_space,
    metric='mean_absolute_error',
    aggregate_metric=['weighted_average', 'average', 'pooling'],  # Select which to report
)

Tip: Use 'pooling' when you care about overall accuracy regardless of which series contributes the error. Use 'average' when all series are equally important regardless of their length.

Custom Metrics

Any callable with signature func(y_true, y_pred) can be passed as a metric. If your metric also needs training data, include a y_train parameter:

# Custom metric without y_train
def mean_bias_error(y_true, y_pred):
    return np.mean(y_pred - y_true)

# Custom metric WITH y_train (detected automatically)
def relative_mase(y_true, y_pred, y_train):
    naive_mae = np.mean(np.abs(np.diff(y_train)))
    return np.mean(np.abs(y_true - y_pred)) / naive_mae

# Both work directly in backtesting
metric, predictions = backtesting_forecaster(
    forecaster=forecaster,
    y=data['target'],
    cv=cv,
    metric=[mean_bias_error, relative_mase],
)

Skforecast automatically wraps callables via add_y_train_argument. If the function already has a y_train parameter, it will receive the training data. If not, the argument is silently ignored.

Common Mistakes

References

See references/metric-compatibility.md for the complete matrix of all available metrics with their properties, compatible forecasters, and usage guidance.