Amey-Thakur Model ensembling
Combining models beats picking the best one, because different models make different errors and averaging cancels them. Ensembling is how competitions are won on the margin. The one requirement most people miss: the models must be diverse, or the ensemble adds nothing.
Method
- Build diverse base models first. The ensemble's power comes from the models disagreeing in their errors: different algorithms (boosted trees, neural nets, linear), different features, different seeds, different data subsets. Averaging five near-identical models barely helps; averaging genuinely different ones helps a lot. Diversity is the goal, not just more models.
- Start with simple averaging. Average the predictions (for probabilities, often the geometric mean or averaging logits; for rank metrics like AUC, rank-averaging) of your strong models. This trivial ensemble usually gives a real, free gain over the best single model and is robust. Weight by validation performance if some models are clearly stronger.
- Blend with a holdout for learned weights. Instead of equal weights, learn the combination weights on a holdout set (a simple linear model over the base predictions). More powerful than averaging, but needs clean out-of-fold predictions to avoid leaking (the blend must not train on data the base models saw for those predictions).
- Stack for the most gains, carefully. Stacking trains a meta-model on the base models' out-of-fold predictions as features. It extracts the most from an ensemble but is the easiest to leak: the base predictions feeding the meta-model must be genuinely out-of-fold (predicted on data that base model did not train on), or you get a fantasy validation score (see cross-validation, train-test-discipline).
- Keep out-of-fold discipline throughout. Every ensembling method depends on honest out-of-fold predictions. Any point where a model's prediction for a row was influenced by that row's label leaks, and the ensemble's local score lies. This is the single failure that sinks ensembles on the private leaderboard.
- Weigh the gain against the cost. Ensembles add accuracy and subtract simplicity, speed, and maintainability. In competitions the accuracy wins; in production a small single model often beats a giant ensemble once latency, cost, and fragility are counted (see model-deployment, ml-baselines).
Boundaries
- Ensembling amplifies whatever the base models have; it cannot fix a fundamental flaw (leakage in the features, a broken validation scheme) shared across them. Garbage in, ensembled garbage out.
- The gains are marginal relative to features and validation; do not reach for stacking before the single-model foundations are strong (see kaggle-competition-workflow's ordering).
- Heavy ensembles rarely belong in production (latency, cost, operational complexity); reserve them for where the last decimal of accuracy is worth it, like a competition (see managed-vs-selfhosted's cost thinking).