cloudthinker-ai ML Model Code Review Skill
Review ML PR #{{ pr_number }} in {{ repository }} for {{ model_type }} model.
Workflow
Phase 1 — Data Leakage
DATA LEAKAGE CHECK
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Train/test split:
[ ] Data split before any preprocessing
[ ] No information from test set in training
[ ] Temporal splits for time-series data
[ ] Group-aware splits (same entity not in both sets)
[ ] Feature leakage:
[ ] No target-derived features
[ ] No future data used in features
[ ] No features unavailable at inference time
[ ] Label encoding fit only on training data
[ ] Preprocessing leakage:
[ ] Scalers/normalizers fit only on training data
[ ] Imputation values from training data only
[ ] Feature selection based on training data only
Phase 2 — Reproducibility
REPRODUCIBILITY CHECK
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Random seeds:
[ ] Random seeds set for all stochastic operations
[ ] Seeds documented and versioned
[ ] Deterministic mode enabled where possible
[ ] Experiment tracking:
[ ] Hyperparameters logged (MLflow, W&B, etc.)
[ ] Training metrics tracked
[ ] Model artifacts versioned
[ ] Data version recorded (DVC, dataset hash)
[ ] Environment:
[ ] Dependencies pinned (requirements.txt / conda.yml)
[ ] GPU/CUDA version documented
[ ] Docker image versioned for training environment
Phase 3 — Bias and Fairness
BIAS ASSESSMENT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Data bias:
[ ] Training data representative of target population
[ ] Class imbalance addressed (oversampling, weights)
[ ] Protected attributes identified
[ ] Demographic parity evaluated
[ ] Model bias:
[ ] Performance metrics computed per subgroup
[ ] Disparate impact ratio calculated
[ ] Fairness constraints applied if needed
[ ] Bias mitigation documented
[ ] Documentation:
[ ] Model card created/updated
[ ] Known limitations documented
[ ] Intended use cases specified
Phase 4 — Model Quality
MODEL QUALITY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Evaluation:
[ ] Appropriate metrics for task (not just accuracy)
[ ] Evaluation on held-out test set
[ ] Cross-validation where appropriate
[ ] Baseline comparison provided
[ ] Production readiness:
[ ] Model size acceptable for deployment target
[ ] Inference latency within SLA
[ ] Model serialization format compatible
[ ] Input validation at inference time
[ ] Graceful handling of missing/malformed features
[ ] A/B test plan for model rollout
Counter-Rationalizations
| Shortcut | Counter | Why |
|---|---|---|
| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |
| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |
| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |
| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |
| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |
Output Format
Produce an ML review report with:
- Data leakage findings (none / potential / confirmed)
- Reproducibility assessment (reproducible / partially / not reproducible)
- Bias and fairness evaluation
- Model quality metrics vs baseline
- Production readiness checklist status