arbazkhan971 Activate When
/godmode:ml, "train a model", "compare experiments"- "evaluate model", "check for bias", "dataset quality"
- ML-related code detected (training loops, features)
Workflow
1. Experiment Definition
ID: EXP-<YYYY-MM-DD>-<NNN>
Hypothesis: <what you expect and why>
Objective: <metric to optimize>
Baseline: <current best or naive baseline>
Task: classification|regression|ranking|generation
Framework: PyTorch|TensorFlow|scikit-learn|JAX|XGBoost
# Check for ML frameworks
pip list 2>/dev/null | grep -iE "torch|tensorflow|sklearn"
cat requirements.txt 2>/dev/null | grep -iE "torch|tf"
2. Hyperparameter Management
search:
strategy: grid|random|bayesian|hyperband
space:
learning_rate: [1e-5, 1e-4, 1e-3, 1e-2]
batch_size: [16, 32, 64, 128]
dropout: uniform(0.1, 0.5)
hidden_size: [128, 256, 512, 1024]
trials: <total>
IF trials > 50: use Bayesian or Hyperband (not grid). IF search space > 4 dimensions: use random search minimum.
3. Dataset Validation
Total samples: <N>
Split: train=<N>(<pct>%) / val=<N>(<pct>%) / test=<N>
Quality checks:
Missing values: <count per feature>
Duplicates: <count exact duplicates>
Outliers: <count, method used>
Class balance: <ratio of majority/minority>
IF class imbalance > 10:1: use stratified sampling
- class weights or oversampling. IF missing > 5% for any feature: investigate before imputing.
4. Bias Detection
Protected attributes: <gender, race, age, geography>
Per-attribute:
| Attribute | Group | Samples | Accuracy | FPR | FNR |
IF max_group_accuracy - min_group_accuracy > 5%:
FLAG bias. Investigate feature correlations.
IF FNR disparity > 10% across groups:
BLOCK deployment until mitigated.
5. Training and Tracking
Epoch: <current>/<total>
Training loss: <value> (trend: decreasing|plateau)
Validation loss: <value> (trend)
Primary metric: <value> (best: <val> at epoch <N>)
IF val_loss increases 3 consecutive epochs: early stop. IF train_loss << val_loss (gap > 2x): overfitting.
6. Model Evaluation
Test set: <N samples> (used ONCE for final eval)
Accuracy: <val> Precision: <val> Recall: <val>
F1: <val> AUC-ROC: <val> AUC-PR: <val>
Statistical significance vs baseline:
p=<val> (paired bootstrap, 10K iterations)
IF p > 0.05: improvement not significant, iterate. IF improvement < 1% absolute: likely noise.
7. Experiment Comparison
| Experiment | F1 | AUC | Latency | Size | Params |
Winner selection: best accuracy/latency tradeoff.
8. Commit and Transition
Commit: "ml: EXP-<ID> — <metric>=<value> (<delta>)"
IF best found: -> /godmode:mlops to deploy.
IF bias detected: address before deployment.
Hard Rules
- NEVER evaluate on test set during development.
- EVERY experiment: git SHA, data version, params, seeds.
- NEVER report without comparing to baseline.
- NEVER deploy model failing bias checks.
- ALWAYS test significance (bootstrap, p < 0.05).
TSV Logging
Append .godmode/ml-results.tsv:
timestamp experiment_id model metric baseline result status
Keep/Discard
KEEP if: significant improvement AND bias passes
AND no data leakage.
DISCARD if: no significance OR bias violation
OR leakage found. Log both.
Stop Conditions
STOP when FIRST of:
- Best model beats baseline significantly
- Bias check passes all attributes
- 3 consecutive experiments show no improvement
Autonomous Operation
On failure: git reset --hard HEAD~1. Never pause.
Error Recovery
| Failure | Action |
|---|---|
| Worse than baseline | Check leakage, preprocessing, balance |
| Training diverges | Reduce LR 10x, check NaN, normalize |
| Fails in production | Compare data distributions, check drift |