Mortality Rate Forecasting Eval

mortality-rate-forecasting-eval

Zero-Shot Forecasting Mortality Rates: A Global Study — Petnehazi et al. (2025) (arXiv:2505.13521, 2025)

What this evaluates

Evaluates zero-shot and fine-tuned time series foundation models against traditional statistical and machine learning baselines for predicting age- and country-specific mortality rates over 5, 10, and 20-year horizons.

Datasets

• Global Mortality Rates (50 countries, 111 age groups) — total ?; splits: test (-1)

Metrics

• SMAPE (primary) — range: percent
  • Symmetric Mean Absolute Percentage Error. Calculated as (100/n) * Σ(|F_t - A_t| / ((|A_t| + |F_t|)/2)), reported as a percentage value.

Input / output format

Input: Historical mortality rate time series for a specific country and age group.

Output: Predicted mortality rates for the next 5, 10, or 20 years.

Scoring recipe

def compute_smape(actual, predicted):
    n = len(actual)
    denominator = (np.abs(actual) + np.abs(predicted)) / 2.0
    denominator = np.where(denominator == 0, 1e-9, denominator)
    smape = 100.0 / n * np.sum(np.abs(predicted - actual) / denominator)
    return smape

Common pitfalls

• SMAPE is unbounded and highly sensitive to near-zero mortality rates, which are common for younger age groups and can inflate error metrics.
• The paper reports median SMAPE across country-age pairs rather than a global aggregate, which can mask performance variations across subgroups.
• Practical significance is strictly defined as a ≥5 percentage point difference in median errors, making it difficult to claim superiority for models with small absolute gains.

Evidence (verbatim from paper)

The mean, median, and standard deviation of SMAPE values for the three methods across 50 countries and 111 age groups are shown in Table 2. The difference between the errors of the 2 models is considered statistically significant if the null hypothesis of the Wilcoxon signed-rank test (that the differences are symmetric about zero) is rejected at the 5% significance level. The difference is considered practically significant if the difference between the median errors of the two models is at least 5% points.

Citation

@misc{petnehazi2025zeroshot,
  title={Zero-Shot Forecasting Mortality Rates: A Global Study},
  author={Petnehazi et al. (2025)},
  year={2025},
  note={arXiv:2505.13521}
}

• arXiv: 2505.13521