Model Selection Guide

Model Selection Guide — ML Model Selection Matrix Guide

Optimal model selection and tuning strategies based on problem type, data characteristics, and constraints.

Model Recommendations by Problem Type

Tabular Data

Unstructured Data

XGBoost vs LightGBM vs CatBoost

Hyperparameter Tuning

Optuna Basic Structure

import optuna

def objective(trial):
    params = {
        'n_estimators': trial.suggest_int('n_estimators', 100, 1000),
        'max_depth': trial.suggest_int('max_depth', 3, 10),
        'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3, log=True),
        'subsample': trial.suggest_float('subsample', 0.6, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree', 0.6, 1.0),
        'reg_alpha': trial.suggest_float('reg_alpha', 1e-8, 10.0, log=True),
        'reg_lambda': trial.suggest_float('reg_lambda', 1e-8, 10.0, log=True),
    }
    model = XGBClassifier(**params)
    score = cross_val_score(model, X, y, cv=5, scoring='f1').mean()
    return score

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)

Tuning Priority

LightGBM tuning order:
Stage 1 (High impact): learning_rate, n_estimators, num_leaves
Stage 2 (Medium impact): max_depth, min_child_samples, subsample
Stage 3 (Low impact): reg_alpha, reg_lambda, colsample_bytree
Stage 4 (Fine-tuning): min_split_gain, path_smooth

Cross-Validation Strategies

Ensemble Methods

Stacking

from sklearn.ensemble import StackingClassifier

estimators = [
    ('xgb', XGBClassifier()),
    ('lgbm', LGBMClassifier()),
    ('cat', CatBoostClassifier(verbose=0)),
]
stack = StackingClassifier(
    estimators=estimators,
    final_estimator=LogisticRegression(),
    cv=5
)

Blending Weights

# Optimal weight search
from scipy.optimize import minimize

def objective(weights):
    pred = sum(w * p for w, p in zip(weights, predictions))
    return -f1_score(y_true, pred > 0.5)

result = minimize(objective, x0=[1/3]*3, constraints={'type': 'eq', 'fun': lambda w: sum(w)-1})

Model Selection Decision Tree

Data type?
├── Tabular
│   ├── Rows < 1,000 → Logistic Regression / SVM
│   ├── 1,000 < Rows < 1M → XGBoost / LightGBM
│   └── Rows > 1M → LightGBM (speed priority)
├── Image → CNN (EfficientNet, ViT)
├── Text → Transformer (BERT)
└── Time Series
    ├── Univariate → Prophet / ARIMA
    └── Multivariate → LightGBM (feature-based) / LSTM