model-comparison skill
Compare two or more regression model reports from any skill in the regression pack.
Produces a ModelComparisonReport with Akaike weights, likelihood-ratio tests for
nested pairs, a coefficient comparison chart, and a structured verdict.
Usage
python .agents/skills/model-comparison/scripts/compare.py \
--reports out/ols/report.json out/ridge/report.json out/lasso/report.json \
--names "OLS" "Ridge" "Lasso" \
--output out/comparison/ \
[--alpha 0.05] \
[--dataset-name "tips"]
Required arguments
| Argument |
Description |
--reports |
Space-separated paths to report.json files (≥ 2) |
--names |
Human-readable model names (same order as --reports) |
--output |
Output directory for report.json and report.html |
Optional arguments
| Argument |
Default |
Description |
--alpha |
0.05 |
Significance level for LR tests |
--dataset-name |
"" |
Label shown in the report header |
What the skill does
Ingest — loads each report.json, sniffs the model family (linear, logistic,
ridge, lasso, elasticnet), and derives a ModelEntry with AIC, BIC, n, k, and
primary fit quality (adj-R² for linear family, pseudo-R² for logistic).
Nesting detection — for each pair of same-family, same-outcome models, checks
whether one's feature set is a strict subset of the other's.
LR test — for each nested pair: LR = 2*(ll_full − ll_nested) ~ chi²(df).
Akaike weights — Δ_i = AIC_i − min(AIC), w_i = exp(−Δ_i/2) / Σ exp(−Δ_j/2).
Only computed for models with valid AIC values.
Verdict — one of:
clear_winner — one model has Akaike weight ≥ 0.80 or LR test rejects simpler modelscompetitive_tie — max Δ AIC < 2complementary_strengths — different families or targets; primary metrics closeall_inadequate — all primary metrics below 0.15
Report — HTML with verdict card, models table, Akaike weight bars, Δ AIC bars,
LR test table, coefficient comparison chart, and flags.
Output schema
ModelComparisonReport
├── models: list[ModelEntry]
├── lr_tests: list[LRTestResult]
├── akaike_weights: AkaikeWeights | None
├── verdict: ComparisonVerdict
│ ├── overall: "clear_winner" | "competitive_tie" | "complementary_strengths" | "all_inadequate"
│ ├── recommended_model: str | None
│ ├── headline: str
│ └── rationale: str
├── flags: list[Flag]
│ ├── SAMPLE_SIZE_MISMATCH — models trained on different n
│ ├── CROSS_FAMILY_COMPARISON — families differ; AIC not valid
│ └── NO_FORMAL_COMPARISON — neither Akaike weights nor LR tests available
└── recommendations: list[Recommendation]
Comparison validity rules
| Scenario |
Akaike weights |
LR test |
| Same family, same outcome, different features |
✓ |
✓ (nested pairs) |
| OLS vs OLS+HC3 (same features) |
✓ (equal weight) |
— |
| OLS vs OLS(log-target) |
✗ |
✗ |
| Linear vs logistic |
✗ |
✗ |
| Any vs Ridge/Lasso (same outcome) |
✓ (approx) |
✗ |
References
- Model selection criteria
- Akaike weights
- Cross-family comparison
1---2name: model-comparison3description: model-comparison skill4---5# model-comparison skill67Compare two or more regression model reports from any skill in the regression pack.8Produces a `ModelComparisonReport` with Akaike weights, likelihood-ratio tests for9nested pairs, a coefficient comparison chart, and a structured verdict.1011## Usage1213```bash14python .agents/skills/model-comparison/scripts/compare.py \15 --reports out/ols/report.json out/ridge/report.json out/lasso/report.json \16 --names "OLS" "Ridge" "Lasso" \17 --output out/comparison/ \18 [--alpha 0.05] \19 [--dataset-name "tips"]20```2122### Required arguments2324| Argument | Description |25|----------|-------------|26| `--reports` | Space-separated paths to `report.json` files (≥ 2) |27| `--names` | Human-readable model names (same order as `--reports`) |28| `--output` | Output directory for `report.json` and `report.html` |2930### Optional arguments3132| Argument | Default | Description |33|----------|---------|-------------|34| `--alpha` | `0.05` | Significance level for LR tests |35| `--dataset-name` | `""` | Label shown in the report header |3637## What the skill does38391. **Ingest** — loads each report.json, sniffs the model family (linear, logistic,40 ridge, lasso, elasticnet), and derives a `ModelEntry` with AIC, BIC, n, k, and41 primary fit quality (adj-R² for linear family, pseudo-R² for logistic).42432. **Nesting detection** — for each pair of same-family, same-outcome models, checks44 whether one's feature set is a strict subset of the other's.45463. **LR test** — for each nested pair: `LR = 2*(ll_full − ll_nested) ~ chi²(df)`.47484. **Akaike weights** — `Δ_i = AIC_i − min(AIC)`, `w_i = exp(−Δ_i/2) / Σ exp(−Δ_j/2)`.49 Only computed for models with valid AIC values.50515. **Verdict** — one of:52 - `clear_winner` — one model has Akaike weight ≥ 0.80 or LR test rejects simpler models53 - `competitive_tie` — max Δ AIC < 254 - `complementary_strengths` — different families or targets; primary metrics close55 - `all_inadequate` — all primary metrics below 0.1556576. **Report** — HTML with verdict card, models table, Akaike weight bars, Δ AIC bars,58 LR test table, coefficient comparison chart, and flags.5960## Output schema6162```63ModelComparisonReport64├── models: list[ModelEntry]65├── lr_tests: list[LRTestResult]66├── akaike_weights: AkaikeWeights | None67├── verdict: ComparisonVerdict68│ ├── overall: "clear_winner" | "competitive_tie" | "complementary_strengths" | "all_inadequate"69│ ├── recommended_model: str | None70│ ├── headline: str71│ └── rationale: str72├── flags: list[Flag]73│ ├── SAMPLE_SIZE_MISMATCH — models trained on different n74│ ├── CROSS_FAMILY_COMPARISON — families differ; AIC not valid75│ └── NO_FORMAL_COMPARISON — neither Akaike weights nor LR tests available76└── recommendations: list[Recommendation]77```7879## Comparison validity rules8081| Scenario | Akaike weights | LR test |82|----------|----------------|---------|83| Same family, same outcome, different features | ✓ | ✓ (nested pairs) |84| OLS vs OLS+HC3 (same features) | ✓ (equal weight) | — |85| OLS vs OLS(log-target) | ✗ | ✗ |86| Linear vs logistic | ✗ | ✗ |87| Any vs Ridge/Lasso (same outcome) | ✓ (approx) | ✗ |8889## References9091- [Model selection criteria](references/criteria.md)92- [Akaike weights](references/akaike_weights.md)93- [Cross-family comparison](references/cross_family.md)
WindcleaverDev