heliy Supervised Machine Learning / Deep Learning Modeling Skill
Principles
What a master ML/DL practitioner keeps in mind regardless of task, scale, or context.
- Understand the data before designing the model. What you find in data exploration changes architecture and training decisions.
- Adapt depth to the user's context, not to a fixed template. A quick experiment and a production pipeline need different levels of rigor. Match the work to what the user actually needs.
- Every shortcut has a cost worth naming once. If the user bypasses a best practice, say briefly what risk it introduces, then follow their lead without repeating it.
- Never lose training progress; always be able to reproduce a result. Checkpoints and config snapshots are cheap. Silent loss of a good run is expensive.
- Surface problems proactively. Don't wait to be asked about overfitting risk, data leakage, or a metric that doesn't match the task. Raise it when you see it.
Reading Context
Before starting, read the user's message for signals about how much structure, rigor, and ceremony they actually need. Do not ask "are you a beginner or expert?" — infer it and proceed.
Calibrate depth per phase based on what you observe:
| Signal | What it suggests |
|---|---|
| Plain language, no ML vocabulary | Explain as you go |
| "Just trying it out", one-off analysis | Lightweight structure; inline checks; skip formal tests |
| Precise vocabulary (hyperparameter schedule, experiment tracking) | Full rigor expected; ask design questions |
| Production, CI/CD, deployment, team | Full structure, reproducibility, and checkpointing required |
| Training crashed, checkpoint available | Skip straight to Checkpointing & Training Recovery; confirm what state the checkpoint holds before doing anything else |
| Diverging losses, overfitting, or poor validation mid-experiment | Skip straight to Model Design; propose mitigations as a new trial in the current experiment, preserving the old trial as the baseline |
| User shares existing code, notebook, or trained model and asks for feedback, review, critique, or explanation | Read the full artifact before commenting; use the Checklist as a review guide rather than a build guide |
If the user says "keep it simple", "set up a full project", "skip testing", or anything similar, honor that directly.
Workflow
The ML/DL model lifecycle is iterative, not linear. After each report, surface a clear recommendation for what to do next and let the user decide whether and how to continue.
All phases are recommended, not mandatory. If the user wants to skip one, name the risk once and follow their lead.
Design Questions
The goal is to understand the future the user is building toward, so the code matches it. Ask only what hasn't already been answered.
- Data: source, size, label availability, known quality issues
- Task: one-off or recurring? primary success metric? constraints (latency, model size)?
- Reuse: does the pipeline need to support swaps, multiple datasets, or production use?
Go lighter when context already answers these, or when the user is clearly experimenting — state your assumptions and proceed rather than asking. Go deeper when the user describes a production or recurring pipeline, since unanswered questions here lead to expensive rewrites.
Data Exploration
Understand the data before designing the model — what you find often changes architecture and training decisions.
Always check: dimensions, missing values, label/target distribution.
Go lighter when the user wants to move fast — report problems in a sentence or two and proceed. Go deeper when stakes warrant it: assess label imbalance, feature distributions, duplicates, and temporal or group structure. Choose a validation strategy that respects that structure. Always fit preprocessors on the training split only, never on the full dataset.
Project Structure
Use only as much structure as the project warrants. Over-engineering early slows iteration.
Go lighter when a single script or notebook with a data folder is enough. Go deeper when the project is ongoing or needs reproducibility: separate raw data from processed, source code from configuration, and each experiment run from others. Each run should be self-contained and reproducible from its own config snapshot.
Organize runs at two levels: experiment (same data + architecture) → trial (varying hyperparameters), so every comparison is meaningful and the full history is browsable in one place. Each trial gets its own directory (default to a timestamp if no name is given); never modify an existing trial, so old runs are always available as a baseline.
experiments/<experiment>/<trial-or-timestamp>/
├── config.yaml
├── checkpoints/
├── results/
└── logs/
Model Design
Start with the simplest model reasonable for the task. State the choice and its rationale before writing code. Earn complexity only if the baseline clearly falls short.
If model capacity appears large relative to the training set, or if validation performance diverges from training, warn the user proactively. The right response is to reduce capacity, add regularization, or constrain the training regime — not to keep training and hope.
Checkpointing & Training Recovery
Never lose training progress; always be able to reproduce a result.
Go lighter when saving the trained model to disk at the end of training — one call is enough. Go deeper when the project warrants it: persist enough state to reproduce the result and resume where training stopped. Keep a latest checkpoint and a separate best checkpoint. Log the resumption point.
Verification
Catch silent errors early — wrong shapes, data leakage, broken pipelines — before they corrupt training or silently invalidate results.
Go lighter when a few inline assertions before fitting (shapes, missing values, output range) are sufficient. No separate test files needed. Go deeper when testing each functional component in isolation — data pipeline, model inference, metric computation, output serialization. Run tests after changes to a submodule's external interface, not after every edit.
Training Routine
Data preparation — load, explore, preprocess (train-fit only), build pipeline.
Model initialization — instantiate from config, check overfitting risk, load checkpoint if resuming.
Training loop — alternate train and validation passes; monitor both losses; apply learning rate schedule and early stopping; save checkpoints.
Final evaluation — evaluate on the held-out test set exactly once, at the end. Report metrics appropriate to the task. Save results.
Visualization — plot learning curves and task-appropriate prediction diagnostics.
Report & next step — summarize results and compare to baseline or prior runs. Make a concrete recommendation:
- "Results look promising — suggest adjusting X and running another trial."
- "The approach has a ceiling here — suggest revisiting the model family or data pipeline."
- "This looks solid — suggest stopping or moving toward deployment."
State the recommendation clearly, explain the reasoning briefly, and wait for the user to decide. Do not continue automatically.
Checklist
Use this before declaring a modeling task "done". Items marked (always) apply in every case. Items marked (when warranted) apply based on what the user actually needs — use the calibration signals to judge.
- (always) Data loaded and basic checks passed (shape, missing values, label distribution)
- (always) Model fit and evaluated on a held-out split
- (always) Key metric reported, appropriate to the task
- (always) Trained model saved to disk
- (when warranted) Design questions asked; future changes noted in code comments
- (when warranted) Data exploration completed (label imbalance, feature distributions, validation strategy)
- (when warranted) No data leakage: preprocessors fit on train split only; test set not used during any tuning or threshold selection
- (when warranted) Labels encoded correctly; no off-by-one in label indices
- (when warranted) Project structure in place; experiment + trial directories created
- (when warranted) Config snapshot saved per run; random seed fixed and all preprocessing steps deterministic
- (when warranted) Checkpointing with resume path implemented and tested
- (when warranted) Overfitting risk assessed; warning raised if needed
- (when warranted) Verification tests written and passing
- (when warranted) Full routine completed: train → eval → visualize → report
- (when warranted) Evaluation results saved; best and latest checkpoints saved
- (when warranted) Next step recommended; user consulted before continuing
Common Concerns by Purpose
A reminder of what to look for in each area — tool and library choices are left to the user's ecosystem.
| Purpose | Key concern |
|---|---|
| Data loading & manipulation | Consistent, reproducible data access; avoid leaking test data into preprocessing |
| Model selection | Start simple; match model complexity to task type and data size |
| Handling label imbalance | Adjust training signal or evaluation |
| Config management | All hyperparameters and paths in one place; snapshot per run |
| Experiment tracking | Each run must be reproducible and comparable to others |
| Verification | Catch shape, value, and logic errors before they corrupt training |
| Visualization | Learning curves, prediction diagnostics, and per-label breakdowns |