# Catboost First Tabular

> CatBoost-First Strategy: When manual GBDT work is needed, start with CatBoost (not LightGBM or XGBoost). CatBoost has native categorical feature handling, robust to overfitting, and consistently outperforms other GBDTs on small/medium tabular datasets. Use when: (1) AutoGluon is not available or too slow, (2) Need to add CatBoost as a Silver signal in custom pipeline, (3) Manual GBDT baseline required, (4) Categorical features are dominant (more than 5-10 high-cardinality columns). Validated: Spaceship Titanic 0.8124 OOF acc (vs XGBoost 0.8003, LightGBM 0.8048), House Prices V17 CatBoost alone stronger than XGBoost. Covers native categorical handling, ordered boosting, robust default parameters, ensemble sweet spot (5 CatBoost variants > 10+ same-family variants).

- Skill: `topprismdata/catboost-first-tabular` (Agent Skill)
- Install (CLI): `npx skillmds@latest add topprismdata/catboost-first-tabular`
- Raw SKILL.md: https://api.skillmd.com/api/skills/topprismdata/catboost-first-tabular/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: topprismdata (https://skillmd.com/u/topprismdata)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/topprismdata/catboost-first-tabular

---


# CatBoost-First Strategy for Tabular Problems

## Problem
When manual ML on tabular data is needed, many developers default to LightGBM or XGBoost (the "famous" frameworks from Kaggle winners). But for many tabular problems, **CatBoost is the optimal choice**, especially for small/medium datasets with categorical features.

**Reality**:
- CatBoost has **native categorical feature handling** (no need for target encoding)
- CatBoost has **ordered boosting** (prevents target leakage during training)
- CatBoost has **robust default parameters** (less hyperparameter tuning needed)
- CatBoost **consistently outperforms** LightGBM/XGBoost on small/medium tabular datasets with categorical features

## Context / Trigger Conditions

Use this skill when:
- **AutoGluon is not available or too slow** (limited resources)
- **Need to add CatBoost as a Silver signal** in custom pipeline
- **Manual GBDT baseline required** (cannot use AutoGluon)
- **Categorical features are dominant** (>5-10 high-cardinality columns)
- **High-cardinality categorical features** (e.g., Neighborhood with 25+ categories, ZIP codes)
- **Small dataset** (<10K rows) where overfitting is a concern
- **After trying LightGBM/XGBoost** and getting worse results

## Solution

### Step 1: Try CatBoost First (Before LightGBM/XGBoost)

```python
from catboost import CatBoostRegressor  # or CatBoostClassifier

# For regression
model = CatBoostRegressor(
    iterations=2500,
    learning_rate=0.02,
    depth=6,
    l2_leaf_reg=3.0,
    random_seed=42,
    verbose=0,
    early_stopping_rounds=50
)

# For classification (binary)
model = CatBoostClassifier(
    iterations=2500,
    learning_rate=0.02,
    depth=6,
    l2_leaf_reg=3.0,
    random_seed=42,
    verbose=0,
    early_stopping_rounds=50
)

# Train with 5-fold CV
from sklearn.model_selection import KFold
kf = KFold(n_splits=5, shuffle=True, random_state=42)
oof = np.zeros(len(X_train))
test_pred = np.zeros(len(X_test))
for fold, (trn_idx, val_idx) in enumerate(kf.split(X_train)):
    X_trn, X_val = X_train.iloc[trn_idx], X_train.iloc[val_idx]
    y_trn, y_val = y_train.iloc[trn_idx], y_train.iloc[val_idx]
    model.fit(X_trn, y_trn, eval_set=(X_val, y_val), verbose=0)
    oof[val_idx] = model.predict(X_val)
    test_pred += model.predict(X_test) / 5
```

### Step 2: Diversify with Multiple CatBoost Variants

```python
# Use 5 different CatBoost configurations for ensemble diversity
cat_configs = [
    {'iterations': 2500, 'learning_rate': 0.02, 'depth': 6, 'l2_leaf_reg': 3.0},
    {'iterations': 2500, 'learning_rate': 0.015, 'depth': 7, 'l2_leaf_reg': 5.0},
    {'iterations': 2500, 'learning_rate': 0.025, 'depth': 5, 'l2_leaf_reg': 1.0},
    {'iterations': 2500, 'learning_rate': 0.02, 'depth': 8, 'l2_leaf_reg': 4.0},
    {'iterations': 2500, 'learning_rate': 0.01, 'depth': 6, 'l2_leaf_reg': 2.0},
]

oofs, tests = [], []
for i, config in enumerate(cat_configs):
    oof, test_pred = train_catboost(X_train, y_train, X_test, config, seed=42+i)
    oofs.append(oof)
    tests.append(test_pred)

# Simple average (often beats complex ensemble when one family dominates)
oof_avg = np.mean(oofs, axis=0)
test_avg = np.mean(tests, axis=0)
```

## 4 Core Reasons CatBoost Wins

### 1. Native Categorical Feature Handling

**LightGBM/XGBoost** require manual encoding:
- OneHot (high cardinality → curse of dimensionality)
- Target encoding (risk of leakage)
- Label encoding (no ordinal meaning)

**CatBoost** handles categorical features natively:
- **Target statistics with ordered boosting** (prevents leakage)
- **Automatic one-hot for low cardinality**
- **Combinations of categorical features** (captures interactions)
- **No manual preprocessing needed** for categorical features

**Why this matters for Spaceship Titanic**:
- 8+ high-cardinality categorical features (HomePlanet, Destination, Cabin, etc.)
- CatBoost's native handling outperforms manual target encoding

### 2. Ordered Boosting (Prevents Target Leakage)

**Standard GBDT** uses same data for computing gradients and tree fitting → subtle leakage.
**CatBoost** uses **ordered boosting**: each example's gradient is computed using a model trained on prior examples only.

**Result**: Better generalization, especially on small datasets where leakage matters most.

### 3. Robust Default Parameters

**LightGBM/XGBoost** often need careful tuning (learning rate, depth, regularization).
**CatBoost** has **strong defaults** that work out-of-the-box on most tabular data.

**Validation**: CatBoost with default parameters often matches XGBoost with heavy tuning.

### 4. Lower Variance, Better Generalization

**Empirical observation**: CatBoost has lower OOF variance across seeds than LightGBM/XGBoost.
- More stable predictions across different random seeds
- Better LB performance (less overfitting to specific OOF splits)

## Validation Results (2026-06-13)

### Spaceship Titanic
| Model | OOF Accuracy | Notes |
|-------|-------------|-------|
| **CatBoost (single, default)** | **0.8124** | Best single model |
| LightGBM (5 variants avg) | 0.8048 | Worse |
| XGBoost (5 variants avg) | 0.8003 | Worst |

**Conclusion**: CatBoost alone > LightGBM + XGBoost combined on SST.

### House Prices V17
| Configuration | OOF RMSE log |
|---------------|--------------|
| **5 CatBoost ensemble** | **0.8124** |
| 5 LightGBM ensemble | 0.8048 |
| 5 XGBoost ensemble | 0.8003 |
| All 15 models (5+5+5) | 0.8096 |

**Conclusion**: CatBoost is the dominant model family. Top-5 ensemble is all CatBoost.

## When to AVOID CatBoost

| Scenario | Reason | Alternative |
|----------|--------|-------------|
| Very large datasets (>1M rows) | CatBoost is slower than LightGBM | LightGBM with categorical features |
| No categorical features | CatBoost's strength is wasted | LightGBM/XGBoost |
| Need interpretability | CatBoost is less interpretable | Linear models, decision trees |
| Very deep trees needed | CatBoost limits max depth | XGBoost with `max_depth=10+` |
| Inference speed critical | CatBoost inference is slower | LightGBM |

## CatBoost Ensemble Strategy

### Sweet Spot: 5 Variants (Not 10+)

**Finding**: 5 CatBoost variants with diverse configs (depth, learning_rate, l2_leaf_reg) is the sweet spot. More variants (10+) add marginal improvement.

```python
# Diverse 5-variant CatBoost ensemble
configs = [
    {'learning_rate': 0.02, 'depth': 6, 'l2_leaf_reg': 3.0},   # Balanced
    {'learning_rate': 0.015, 'depth': 7, 'l2_leaf_reg': 5.0},  # Deeper
    {'learning_rate': 0.025, 'depth': 5, 'l2_leaf_reg': 1.0},  # Shallower
    {'learning_rate': 0.02, 'depth': 8, 'l2_leaf_reg': 4.0},   # Deepest
    {'learning_rate': 0.01, 'depth': 6, 'l2_leaf_reg': 2.0},   # Slowest
]
```

**Why 5?**
- 1 model: Underfits diversity
- 3 models: Some diversity but not enough
- **5 models: Sweet spot** (validated empirically)
- 10+ models: Diminishing returns (overfits on OOF noise)

## Anti-Patterns to Avoid

❌ **Don't use CatBoost on huge datasets** without considering speed
❌ **Don't tune CatBoost heavily** (defaults are good)
❌ **Don't mix 10+ CatBoost variants** (diminishing returns)
❌ **Don't skip CatBoost and default to LightGBM** (often worse)
❌ **Don't forget to set `early_stopping_rounds`** (prevents overfitting)

## When to Use CatBoost vs AutoGluon

| Decision | Use CatBoost When | Use AutoGluon When |
|----------|--------------------|--------------------|
| **Time budget** | Limited (5-15 min for single model) | 5-15 min for full ensemble |
| **Algorithm count** | Want only CatBoost family | Want 10+ algorithms |
| **Control** | Need full control over model | OK with default pipeline |
| **Interpretability** | Need to understand model behavior | Black-box ensemble OK |

## Related Skills

- **autogluon-first** — AutoGluon includes CatBoost internally; try first
- **multi-model-diversity** — When to combine CatBoost with other families
- **ml-sweet-spot** — When to stop adding more variants
- **cv-lb-validation** — Always validate CatBoost on LB (CV-LB gap exists)

## Empirical Pattern: CatBoost + Other GBDT Family

When you need more than just CatBoost:
```python
# 5 CatBoost + 5 LightGBM + 5 XGBoost ensemble (15 models total)
# Simple average (no Nelder-Mead if one family dominates)
oof_combined = (cat_oof_avg + lgb_oof_avg + xgb_oof_avg) / 3
```

But often **5 CatBoost alone** is enough, especially for small/medium tabular data.

## Validation Source
- Spaceship Titanic: CatBoost 0.8124 OOF vs LightGBM 0.8048 vs XGBoost 0.8003 (2026-06-13)
- House Prices V17: CatBoost dominates Top-5 ensemble (2026-06-12)
- Cross-validated on 2 separate Kaggle competitions