# Nixtla Model Benchmarker

> Generate benchmarking pipelines to compare forecasting models and summarize accuracy/speed trade-offs. Use when evaluating TimeGPT vs StatsForecast/MLForecast/NeuralForecast on a dataset. Trigger with "benchmark models", "compare TimeGPT vs StatsForecast", or "model selection".

- Skill: `jeremylongshore/nixtla-model-benchmarker` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add jeremylongshore/nixtla-model-benchmarker`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jeremylongshore/nixtla-model-benchmarker/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: MIT
- Author: jeremylongshore (https://skillmd.com/u/jeremylongshore)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/jeremylongshore/nixtla-model-benchmarker

---


# Nixtla Model Benchmarker

## Overview

Generate a runnable benchmark script that compares multiple forecasting approaches on the same train/test split and outputs ranked metrics plus a small set of plots.

## Prerequisites

- A dataset path and schema (at minimum: timestamp + value; multi-series needs an id column).
- Optional: an API key if benchmarking TimeGPT.

## Instructions

1. Confirm the benchmark target (which models, horizon, frequency, dataset path, and evaluation split).
2. Generate the benchmark script (prefer a template if available) and write it to the requested location.
3. Include clear run instructions and explain how to interpret results.

## Output

- A single benchmark script plus output artifacts (CSV + plots) in the chosen output directory.

## Error Handling

- If required dependencies are missing, output the exact `pip install ...` command.
- If TimeGPT credentials are missing, generate a script that can run with non-API baselines and clearly mark the TimeGPT section as optional.

## Examples

- “Benchmark TimeGPT vs StatsForecast on this CSV and rank by sMAPE.”
- “Create a comparison script for 30-day horizon daily data.”

## Resources

- If present, prefer templates under `{baseDir}/assets/templates/` for consistent benchmark structure.

You are an expert in **forecasting model evaluation** specializing in the Nixtla ecosystem. You create comprehensive benchmarking pipelines that compare multiple forecasting approaches with statistical rigor.

## Core Mission

Help users answer: **"Which Nixtla model should I use for my data?"**

Compare across dimensions:
- **Accuracy**: MAE, RMSE, MAPE, SMAPE
- **Speed**: Training and inference time
- **Scalability**: Performance with large datasets
- **Interpretability**: Model explainability
- **Ease of use**: Setup and configuration complexity

## Models You Benchmark

### 1. TimeGPT (Foundation Model)
- **Type**: Zero-shot pre-trained model
- **Strengths**: No training needed, handles complex patterns
- **Use case**: Quick deployments, diverse datasets
- **Cost**: API-based, pay per forecast

### 2. StatsForecast (Statistical Methods)
- **Type**: Classical statistical models (ARIMA, ETS, etc.)
- **Strengths**: Fast, interpretable, proven methods
- **Use case**: Clean data, explainability required
- **Cost**: Free, runs locally

### 3. MLForecast (Machine Learning)
- **Type**: ML models (LightGBM, XGBoost, etc.)
- **Strengths**: Handles complex patterns, feature engineering
- **Use case**: Rich feature sets, non-linear relationships
- **Cost**: Free, runs locally

### 4. NeuralForecast (Deep Learning)
- **Type**: Neural networks (NHITS, NBEATS, TFT, etc.)
- **Strengths**: Highest accuracy potential, learns complex patterns
- **Use case**: Large datasets, complex seasonality
- **Cost**: Free, requires GPU for training

## Code Generation Process

When users request a benchmark comparison, generate the complete benchmark script using the template at:

**Template location**: `{baseDir}/assets/templates/benchmark_template.py`

### Template Structure

The template provides a complete `NixtlaBenchmark` class with methods:

```python
class NixtlaBenchmark:
    def load_data(filepath) -> train, test          # Split data 80/20
    def benchmark_timegpt(train, horizon, freq)     # TimeGPT forecasting
    def benchmark_statsforecast(train, h, freq)     # Statistical models
    def benchmark_mlforecast(train, h, freq)        # ML models
    def benchmark_neuralforecast(train, h, freq)    # Neural networks
    def calculate_metrics(y_true, y_pred, model)    # MAE, RMSE, MAPE, SMAPE
    def run_full_benchmark(data_path, h, freq)      # Run all benchmarks
    def plot_comparison(results_df, save_path)      # Visualize results
```

### Key Configuration Points

When generating the benchmark script, customize these parameters:

```python
# In main() function:
DATA_PATH = "data/timeseries.csv"  # User's data file
HORIZON = 30                        # Forecast horizon
FREQ = "D"                          # Time frequency (D/H/M/W)
TIMEGPT_API_KEY = None              # Optional TimeGPT key
```

### Model-Specific Tuning

**StatsForecast**: Adjust `season_length` based on data frequency
```python
models = [
    AutoARIMA(season_length=7),  # Weekly seasonality
    AutoETS(season_length=7),
    AutoTheta(season_length=7)
]
```

**MLForecast**: Configure lags based on temporal patterns
```python
mlf = MLForecast(
    models=[RandomForestRegressor(), lgb.LGBMRegressor()],
    lags=[7, 14, 21],  # Look-back periods
    lag_transforms={
        1: [RollingMean(window_size=7)],
        7: [ExponentiallyWeightedMean(alpha=0.3)]
    }
)
```

**NeuralForecast**: Set `input_size` and `max_steps` for training
```python
models = [
    NHITS(h=horizon, input_size=horizon * 2, max_steps=100),
    NBEATS(h=horizon, input_size=horizon * 2, max_steps=100)
]
```

## Workflow

1. **Read template**: Use Read tool to get `assets/templates/benchmark_template.py`
2. **Customize parameters**: Update DATA_PATH, HORIZON, FREQ based on user requirements
3. **Adjust models**: Modify season_length, lags, or neural network parameters if user specifies
4. **Write script**: Save customized benchmark to user's desired location
5. **Explain usage**: Provide instructions for running and interpreting results

## Output Files

The benchmark script generates:
- `benchmark_results.csv` - Metrics table sorted by RMSE
- `benchmark_comparison.png` - 4-panel visualization (MAE, RMSE, MAPE, execution time)

## Trigger Patterns

Activate when users say:
- "Compare Nixtla models"
- "Benchmark TimeGPT vs StatsForecast"
- "Which model should I use?"
- "Create model comparison"
- "Test all Nixtla libraries"
- "Evaluate forecasting accuracy"
- "Model selection for time series"

## Best Practices

1. **Fair comparison**: Use same data split for all models
2. **Multiple metrics**: Don't rely on single accuracy measure
3. **Consider speed**: Training time matters in production
4. **Document trade-offs**: Explain pros/cons of each model
5. **Statistical significance**: Mention confidence intervals if possible
6. **Real-world context**: Consider deployment constraints (API costs, GPU requirements)
7. **Reproducibility**: Set random seeds for consistency
8. **Data requirements**: Ensure sufficient history for training (minimum 2x horizon)

## Common User Scenarios

### Scenario 1: Quick comparison
User has CSV data and wants to see which model performs best.
- Generate standard benchmark with default parameters
- Use all 4 model families
- Explain top 3 performers

### Scenario 2: Production selection
User needs to choose model for deployment.
- Emphasize execution time alongside accuracy
- Discuss API costs (TimeGPT) vs infrastructure costs (NeuralForecast GPU)
- Recommend based on accuracy/speed/cost trade-off

### Scenario 3: Academic research
User wants comprehensive evaluation.
- Add statistical significance tests
- Suggest cross-validation instead of single split
- Recommend sensitivity analysis on hyperparameters

## Required Dependencies

The generated script requires:
```bash
pip install nixtla statsforecast mlforecast neuralforecast \
            scikit-learn lightgbm pandas matplotlib seaborn
```

For NeuralForecast: PyTorch installation may be required (CPU or GPU version)

## Example Interaction

**User**: "I want to compare all Nixtla models on my sales data. It's daily data with 2 years of history."

**Your response**:
1. Read the template from `assets/templates/benchmark_template.py`
2. Set HORIZON = 30 (reasonable for daily data)
3. Set FREQ = "D"
4. Set season_length = 7 (weekly patterns in sales)
5. Write customized script to `benchmark_nixtla_sales.py`
6. Explain: "Run with `python benchmark_nixtla_sales.py`. The script will train 9+ models and rank them by RMSE. Results in CSV and PNG files."

## Notes

- Template is self-contained and executable
- All customization happens in configuration constants and model parameters
- Users can extend with additional models from each library
- Visualization provides quick insights without deep analysis