Autogluon Preset Strategy

AutoGluon Preset Strategy

Problem

autogluon-first says "use best_quality preset always". But:

• best_quality takes 1-4 hours — wasted if you only need a quick baseline
• medium_quality is the default but doesn't enable bagging/stacking → no OOF predictions
• The choice between presets has a 5x time × accuracy tradeoff that the existing skill doesn't quantify

Decision Tree

N = training rows
N < 10K   → presets='best_quality' (single GBDT plateau is real concern)
N 10K-100K → presets='good_quality' first (10-30 min), upgrade to high if time allows
N > 100K  → presets='good_quality' is enough (data variance dominates model choice)

GPU available? 
  No → presets='high_quality' max (TabPFNv2 needs GPU)
  Yes → consider 'extreme_quality' (uses TabPFNv2, TabICL, Mitra)

Metric?
  balanced_accuracy / F1 → add calibrate_decision_threshold=True
  accuracy → default ('auto')
  log_loss → use predict_proba, no calibration

Preset Reference (from AG 1.4.0 docstring)

Why medium_quality has no OOF: AG's predict_oof() requires bag mode (auto_stack=False means no bag). The "val score" reported is in-sample on the holdout — not real OOF. Don't trust it.

Empirical Validation (s6e7, N=690K, 3-class)

Surprising finding: medium_quality's val score (0.8825) is higher than good/high (0.873). This is fake: medium has no bag/stack, val is in-sample on the internal holdout. The true OOF (good=0.8730, high=0.8739) is what you should trust.

Practical implication: good→high gives only +0.0009 OOF improvement at 2.5x time cost. good_quality is the sweet spot unless you have hours to burn.

The "Do I Need EDA?" Question

Short answer: For standard tabular (numeric + categorical features, no time series, no text), you do NOT need EDA before fitting AG.

Validation: s6e7 (690K rows, 14 features, ~50% missing) — fit AG directly with no manual EDA. Result: OOF=0.8739, LB=0.87458, gap=0.0007. No EDA was needed.

Tuning Within a Preset

After baseline (good_quality, 30 min), try in order:

Priority 1: Increase time_limit (cheapest)

predictor.fit(tr, time_limit=3600, presets='good_quality')  # 6x longer

Priority 2: Customize key models

predictor.fit(tr, presets='good_quality',
              hyperparameters={'GBM': {'num_leaves': 64, 'learning_rate': 0.03},
                              'CAT': {'depth': 8}})

Priority 3: Use refit_full (more data = better)

predictor.fit(tr, presets='high_quality', refit_full=True)
# refit trains on ALL data (no holdout) — typically +0.001-0.005 OOF

Priority 4: Decision threshold calibration

predictor.fit(tr, eval_metric='balanced_accuracy',
              calibrate_decision_threshold=True)
# Helps when metric is balanced_accuracy or F1

Standard Workflow

# Step 1: Baseline (10-30 min)
predictor = TabularPredictor(label='target', eval_metric='accuracy').fit(
    tr, time_limit=600, presets='good_quality')

# Step 2: Inspect (1 min)
print(predictor.leaderboard(silent=True))
print(predictor.feature_importance(tr))

# Step 3: If you need more (decide based on Step 2)
# - Models are similar → refit_full=True
# - One model dominates → tune hyperparameters
# - Need more time → longer time_limit
predictor = TabularPredictor(label='target', eval_metric='accuracy').fit(
    tr, time_limit=3600, presets='good_quality',
    hyperparameters={'GBM': {'num_leaves': 64}})

# Step 4: Predict and submit
pred = predictor.predict(te)

When AG Wins and Loses

AG Wins:

• Standard tabular with 1K-1M rows
• No time budget constraints
• You don't need to understand model decisions
• Multiple data types (numeric + categorical + text) — AG handles all

AG Loses:

• Real-time inference (<10ms latency) — AG ensembles are slow
• Custom loss functions
• Tiny data (<100 rows) — GBDT overfits, AG can't help
• Adversarial / non-stationary targets — AG assumes IID
• You need SHAP/explainability for each model decision

Anti-Patterns

Key API Methods

# Inspect
predictor.leaderboard(silent=True)              # Model rankings + val scores
predictor.feature_importance(tr)                 # Permutation importance
predictor.model_names()                         # List of trained models
predictor.info()                                # Full info

# Predict
predictor.predict(te)                           # Class labels
predictor.predict_proba(te)                     # Class probabilities

# Inspect best model
best_model = predictor.model_best
predictor.predict_oof()                          # Only if bag mode enabled
predictor.refit_full()                          # Retrain best on all data

When to Use This Skill vs autogluon-first

References

• AG 1.4.0 docstring (inspect.getsource on TabularPredictor.fit)
• AutoGluon 1.4.0 release notes
• Erickson et al. (2020) "AutoGluon-Tabular" arXiv:2003.06505
• Salinas & Erickson (2025) "TabRepo" arXiv:2507.07829 (zeroshot hyperparameter portfolio)