Amey-Thakur Hyperparameter tuning
Tuning is compute spent buying metric points. Spend it with a budget, a search strategy better than grid, and the humility that hundreds of trials against one validation set will overfit to that set.
Method
- Tune only after the pipeline is trustworthy. Splits disciplined (see train-test-discipline), features leak-free (see feature-engineering), baseline recorded (see ml-baselines). Tuning a leaky pipeline polishes a lie; and data/feature improvements usually dwarf hyperparameter deltas, so exhaust them first.
- Search the parameters that matter, on the right scales. Learning rate (log scale), regularization strength (log), tree depth/leaves, subsampling rates; leave third-order knobs at defaults. Random or quasi-random search beats grid at equal budget (grid wastes trials on unimportant axes); Bayesian/model-based search (Optuna-class) pays off when single trials are expensive.
- Use early stopping as both tuner and guard. For iterative learners (boosting, neural nets), fix a generous iteration cap and early-stop on validation: it tunes the most sensitive parameter for free. At the search level, successive-halving/Hyperband prunes bad trials at low fidelity so the budget concentrates on contenders.
- Set the budget before the first trial. N trials or X GPU-hours, decided from the metric's value (see cloud-cost-optimization thinking); log every trial's config and result (see experiment-tracking). Open-ended tuning runs until someone gets bored, which is a stopping rule that favors noise.
- Protect the validation set from the search. Hundreds of trials maximize validation-set luck along with skill: use K-fold CV inside the search when data is small (see cross-validation), re-fit the winner on train+validation only per your protocol, and report the final number from the untouched test set once. Expect test to come in slightly under the tuned validation score; if it craters, the search overfit.
- Prefer flat optima and reproducibility. Between configs within noise of each other, take the one in a wide flat region (stable to drift and retrains) over a sharp peak; verify the winner across 2-3 seeds before believing it (see model-evaluation uncertainty). Record final config in code, not in a notebook's memory.
Boundaries
- Tuning cannot rescue the wrong model class or missing signal; a plateaued search is evidence to change representation, not to add trials.
- Percent-level gains that vanish across seeds are noise laundered through effort; the ship decision weighs stability and cost, not leaderboard decimals.
- AutoML earns its keep for breadth on tabular problems; it still obeys every rule here (budget, validation hygiene, test-once), and its winner still needs the deployment review (see model-deployment).