jcorpac ML Hyperparameter Tuning
Tuning is the difference between an "okay" model and a "great" one.
The Search Space
Identify which parameters have the most impact (e.g., n_estimators, learning_rate, max_depth).
Methods
- Grid Search: Exhaustive search over a fixed grid. Reliable but slow.
- Random Search: Random sampling of the space. Often better than grid search for fewer iterations.
- Bayesian Optimization (Optuna): Uses previous trials to inform the next search. The state-of-the-art for ML tuning.
Best Practices
- Pruning: Stop poor-performing trials to save time.
- Nested Cross-Validation: To avoid overfitting the hyperparameters themselves.