# Nixtla Forecast Validator

> Validate forecast quality by comparing MASE and sMAPE against benchmarks. Use when detecting model degradation. Trigger with 'validate forecast' or 'check forecast quality'.

- Skill: `jeremylongshore/nixtla-forecast-validator` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add jeremylongshore/nixtla-forecast-validator`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jeremylongshore/nixtla-forecast-validator/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: MIT
- Author: jeremylongshore (https://skillmd.com/u/jeremylongshore)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/jeremylongshore/nixtla-forecast-validator

---


# Nixtla Forecast Validator

Validates time series forecast quality metrics and detects performance degradation using statistical measures. Compares current forecast accuracy against historical benchmarks to identify significant deviations in MASE and sMAPE metrics.

## Overview

This skill analyzes forecast quality by comparing current performance metrics against historical baselines. It detects significant increases in error metrics (MASE and sMAPE) that may indicate model degradation, data quality issues, or changing patterns in the time series. The skill generates comprehensive reports, alerts, and visualizations to help users identify and address forecast quality problems quickly.

Activates automatically when Claude detects forecast validation needs, or when explicitly requested with phrases like "validate forecast quality", "check model performance", or "assess forecast accuracy".

## Prerequisites

**Tools**: Read, Write, Bash, Glob, Grep

**Environment**: No API keys required (operates on CSV metrics files)

**Python Packages**:
```bash
pip install pandas matplotlib
```

**Required CSV Format**:
CSV files must contain columns: `model`, `MASE`, `sMAPE`

## Instructions

### Step 1: Prepare metrics data

Ensure you have two CSV files containing forecast metrics:
- Historical metrics CSV (baseline performance)
- Current metrics CSV (recent performance to validate)

Each CSV must have columns: `model`, `MASE`, `sMAPE`

**Example format**:
```
model,MASE,sMAPE
model_A,1.2,0.15
model_B,0.8,0.10
```

### Step 2: Set validation thresholds

Configure acceptable deviation thresholds for MASE and sMAPE metrics. Default thresholds are 0.2 (20% increase), but these can be adjusted based on business requirements and model characteristics.

**Recommended thresholds**:
- Conservative: 0.1 (10% increase triggers alert)
- Standard: 0.2 (20% increase triggers alert)
- Lenient: 0.3 (30% increase triggers alert)

### Step 3: Execute validation

Run the validation script to compare current metrics against historical benchmarks:

```bash
python {baseDir}/scripts/validate_forecast.py \
  --historical historical_metrics.csv \
  --current current_metrics.csv \
  --mase_threshold 0.2 \
  --smape_threshold 0.2
```

The script performs:
1. Loads historical and current metrics from CSV files
2. Calculates percentage increase for each metric per model
3. Compares increases against configured thresholds
4. Generates validation report, comparison CSV, alert log, and visualization

### Step 4: Review validation outputs

Analyze the generated outputs to identify forecast quality issues:
- Read `validation_report.txt` for summary of findings
- Check `alert.log` for models requiring immediate attention
- Review `metrics_comparison.csv` for detailed metric changes
- Examine `metrics_visualization.png` for visual comparison

If degradation is detected, investigate potential causes such as data quality changes, concept drift, or model staleness.

## Output

The validation process generates four output files:

1. **validation_report.txt**: Summary report indicating which models show significant degradation and overall validation status
2. **metrics_comparison.csv**: Side-by-side comparison of historical vs current metrics for all models
3. **alert.log**: Alert messages for models exceeding degradation thresholds
4. **metrics_visualization.png**: Bar chart visualization comparing historical and current MASE and sMAPE values

## Error Handling

**Common errors and solutions**:

1. **Missing required metrics column (MASE or sMAPE)**
   - Ensure input CSV files contain columns named exactly `MASE` and `sMAPE` (case-sensitive)
   - Verify column headers match expected format

2. **Invalid threshold value**
   - Provide positive numerical values for `--mase_threshold` and `--smape_threshold`
   - Thresholds represent percentage increase (0.2 = 20%)

3. **Historical data unavailable**
   - Verify path to historical metrics CSV file is correct
   - Ensure file exists and is readable
   - Check file format matches required CSV structure

4. **File not found error**
   - Verify both `--historical` and `--current` file paths are correct
   - Use absolute paths if relative paths fail
   - Check file permissions

5. **Empty DataFrame error**
   - Ensure CSV files are not empty
   - Verify CSV files contain data rows beyond the header
   - Check for proper CSV formatting (commas as delimiters)

## Examples

### Example 1: Significant MASE degradation detected

**Input** (historical_metrics.csv):
```
model,MASE,sMAPE
model_A,1.2,0.15
```

**Input** (current_metrics.csv):
```
model,MASE,sMAPE
model_A,1.8,0.18
```

**Command**:
```bash
python scripts/validate_forecast.py --historical historical_metrics.csv --current current_metrics.csv
```

**Output** (validation_report.txt):
```
WARNING: Significant increase in MASE detected for model model_A.
```

**Interpretation**: Model A shows 50% increase in MASE (from 1.2 to 1.8), exceeding the default 20% threshold. This indicates forecast quality degradation requiring investigation.

### Example 2: Stable performance, no alerts

**Input** (historical_metrics.csv):
```
model,MASE,sMAPE
model_B,0.8,0.10
```

**Input** (current_metrics.csv):
```
model,MASE,sMAPE
model_B,0.85,0.11
```

**Command**:
```bash
python scripts/validate_forecast.py --historical historical_metrics.csv --current current_metrics.csv
```

**Output** (validation_report.txt):
```
Forecast validation passed. No significant degradation detected.
```

**Interpretation**: Model B shows only 6.25% increase in MASE and 10% increase in sMAPE, both below the 20% threshold. Performance is stable.

### Example 3: Multiple models with custom thresholds

**Command**:
```bash
python scripts/validate_forecast.py \
  --historical multi_model_historical.csv \
  --current multi_model_current.csv \
  --mase_threshold 0.3 \
  --smape_threshold 0.25
```

Uses more lenient thresholds (30% for MASE, 25% for sMAPE) suitable for volatile forecasts or experimental models.

## Resources

**Script**: `{baseDir}/scripts/validate_forecast.py`

**Metrics**: MASE (Mean Absolute Scaled Error), sMAPE (symmetric Mean Absolute Percentage Error)

**Related skills**: `nixtla-timegpt-lab`, `nixtla-experiment-architect`, `nixtla-schema-mapper`