Machine Learning
Comprehensive machine learning skill covering the full ML lifecycle from experimentation to production deployment.
When to Use This Skill
- Building machine learning pipelines
- Feature engineering and data preprocessing
- Model training, evaluation, and selection
- Hyperparameter tuning and optimization
- Model deployment and serving
- ML experiment tracking and versioning
- Production ML monitoring and maintenance
ML Development Lifecycle
1. Problem Definition
Classification Types:
- Binary classification (spam/not spam)
- Multi-class classification (image categories)
- Multi-label classification (document tags)
- Regression (price prediction)
- Clustering (customer segmentation)
- Ranking (search results)
- Anomaly detection (fraud detection)
Success Metrics by Problem Type:
| Problem Type |
Primary Metrics |
Secondary Metrics |
| Binary Classification |
AUC-ROC, F1 |
Precision, Recall, PR-AUC |
| Multi-class |
Macro F1, Accuracy |
Per-class metrics |
| Regression |
RMSE, MAE |
R², MAPE |
| Ranking |
NDCG, MAP |
MRR |
| Clustering |
Silhouette, Calinski-Harabasz |
Davies-Bouldin |
2. Data Preparation
Data Quality Checks:
- Missing value analysis and imputation strategies
- Outlier detection and handling
- Data type validation
- Distribution analysis
- Target leakage detection
Feature Engineering Patterns:
- Numerical: scaling, binning, log transforms, polynomial features
- Categorical: one-hot, target encoding, frequency encoding, embeddings
- Temporal: lag features, rolling statistics, cyclical encoding
- Text: TF-IDF, word embeddings, transformer embeddings
- Geospatial: distance features, clustering, grid encoding
Train/Test Split Strategies:
- Random split (standard)
- Stratified split (imbalanced classes)
- Time-based split (temporal data)
- Group split (prevent data leakage)
- K-fold cross-validation
3. Model Selection
Algorithm Selection Guide:
| Data Size |
Problem |
Recommended Models |
| Small (<10K) |
Classification |
Logistic Regression, SVM, Random Forest |
| Small (<10K) |
Regression |
Linear Regression, Ridge, SVR |
| Medium (10K-1M) |
Classification |
XGBoost, LightGBM, Neural Networks |
| Medium (10K-1M) |
Regression |
XGBoost, LightGBM, Neural Networks |
| Large (>1M) |
Any |
Deep Learning, Distributed training |
| Tabular |
Any |
Gradient Boosting (XGBoost, LightGBM, CatBoost) |
| Images |
Classification |
CNN, ResNet, EfficientNet, Vision Transformers |
| Text |
NLP |
Transformers (BERT, RoBERTa, GPT) |
| Sequential |
Time Series |
LSTM, Transformer, Prophet |
4. Model Training
Hyperparameter Tuning:
- Grid Search: exhaustive, good for small spaces
- Random Search: efficient, good for large spaces
- Bayesian Optimization: smart exploration (Optuna, Hyperopt)
- Early stopping: prevent overfitting
Common Hyperparameters:
| Model |
Key Parameters |
| XGBoost |
learning_rate, max_depth, n_estimators, subsample |
| LightGBM |
num_leaves, learning_rate, n_estimators, feature_fraction |
| Random Forest |
n_estimators, max_depth, min_samples_split |
| Neural Networks |
learning_rate, batch_size, layers, dropout |
5. Model Evaluation
Evaluation Best Practices:
- Always use held-out test set for final evaluation
- Use cross-validation during development
- Check for overfitting (train vs validation gap)
- Evaluate on multiple metrics
- Analyze errors qualitatively
Handling Imbalanced Data:
- Resampling: SMOTE, undersampling
- Class weights: weighted loss functions
- Threshold tuning: optimize decision threshold
- Evaluation: use PR-AUC over ROC-AUC
6. Production Deployment
Model Serving Patterns:
- REST API (Flask, FastAPI, TF Serving)
- Batch inference (scheduled jobs)
- Streaming (real-time predictions)
- Edge deployment (mobile, IoT)
Production Considerations:
- Latency requirements (p50, p95, p99)
- Throughput (requests per second)
- Model size and memory footprint
- Fallback strategies
- A/B testing framework
7. Monitoring & Maintenance
What to Monitor:
- Prediction latency
- Input feature distributions (data drift)
- Prediction distributions (concept drift)
- Model performance metrics
- Error rates and types
Retraining Triggers:
- Performance degradation below threshold
- Significant data drift detected
- Scheduled retraining (daily, weekly)
- New training data available
MLOps Best Practices
Experiment Tracking
Track for every experiment:
- Code version (git commit)
- Data version (hash or version ID)
- Hyperparameters
- Metrics (train, validation, test)
- Model artifacts
- Environment (packages, versions)
Model Versioning
models/
├── model_v1.0.0/
│ ├── model.pkl
│ ├── metadata.json
│ ├── requirements.txt
│ └── metrics.json
├── model_v1.1.0/
└── model_v2.0.0/
CI/CD for ML
Continuous Integration:
- Data validation tests
- Model training tests
- Performance regression tests
Continuous Deployment:
- Staging environment validation
- Shadow mode testing
- Gradual rollout (canary)
- Automatic rollback
Reference Files
For detailed patterns and code examples, load reference files as needed:
references/preprocessing.md - Data preprocessing patterns and feature engineering techniquesreferences/model_patterns.md - Model architecture patterns and implementation examplesreferences/evaluation.md - Comprehensive evaluation strategies and metrics
Integration with Other Skills
- performance - For optimizing inference latency
- testing - For ML-specific testing patterns
- database-optimization - For feature store queries
- debugging - For model debugging and error analysis
1---2name: machine-learning3description: Machine learning development patterns, model training, evaluation, and deployment. Use when building ML pipelines, training models, feature engineering, model evaluation, or deploying ML systems to production.4---56# Machine Learning78Comprehensive machine learning skill covering the full ML lifecycle from experimentation to production deployment.910## When to Use This Skill1112- Building machine learning pipelines13- Feature engineering and data preprocessing14- Model training, evaluation, and selection15- Hyperparameter tuning and optimization16- Model deployment and serving17- ML experiment tracking and versioning18- Production ML monitoring and maintenance1920## ML Development Lifecycle2122### 1. Problem Definition2324**Classification Types:**2526- Binary classification (spam/not spam)27- Multi-class classification (image categories)28- Multi-label classification (document tags)29- Regression (price prediction)30- Clustering (customer segmentation)31- Ranking (search results)32- Anomaly detection (fraud detection)3334**Success Metrics by Problem Type:**3536| Problem Type | Primary Metrics | Secondary Metrics |37|--------------|-----------------|-------------------|38| Binary Classification | AUC-ROC, F1 | Precision, Recall, PR-AUC |39| Multi-class | Macro F1, Accuracy | Per-class metrics |40| Regression | RMSE, MAE | R², MAPE |41| Ranking | NDCG, MAP | MRR |42| Clustering | Silhouette, Calinski-Harabasz | Davies-Bouldin |4344### 2. Data Preparation4546**Data Quality Checks:**4748- Missing value analysis and imputation strategies49- Outlier detection and handling50- Data type validation51- Distribution analysis52- Target leakage detection5354**Feature Engineering Patterns:**5556- Numerical: scaling, binning, log transforms, polynomial features57- Categorical: one-hot, target encoding, frequency encoding, embeddings58- Temporal: lag features, rolling statistics, cyclical encoding59- Text: TF-IDF, word embeddings, transformer embeddings60- Geospatial: distance features, clustering, grid encoding6162**Train/Test Split Strategies:**6364- Random split (standard)65- Stratified split (imbalanced classes)66- Time-based split (temporal data)67- Group split (prevent data leakage)68- K-fold cross-validation6970### 3. Model Selection7172**Algorithm Selection Guide:**7374| Data Size | Problem | Recommended Models |75|-----------|---------|-------------------|76| Small (<10K) | Classification | Logistic Regression, SVM, Random Forest |77| Small (<10K) | Regression | Linear Regression, Ridge, SVR |78| Medium (10K-1M) | Classification | XGBoost, LightGBM, Neural Networks |79| Medium (10K-1M) | Regression | XGBoost, LightGBM, Neural Networks |80| Large (>1M) | Any | Deep Learning, Distributed training |81| Tabular | Any | Gradient Boosting (XGBoost, LightGBM, CatBoost) |82| Images | Classification | CNN, ResNet, EfficientNet, Vision Transformers |83| Text | NLP | Transformers (BERT, RoBERTa, GPT) |84| Sequential | Time Series | LSTM, Transformer, Prophet |8586### 4. Model Training8788**Hyperparameter Tuning:**8990- Grid Search: exhaustive, good for small spaces91- Random Search: efficient, good for large spaces92- Bayesian Optimization: smart exploration (Optuna, Hyperopt)93- Early stopping: prevent overfitting9495**Common Hyperparameters:**9697| Model | Key Parameters |98|-------|---------------|99| XGBoost | learning_rate, max_depth, n_estimators, subsample |100| LightGBM | num_leaves, learning_rate, n_estimators, feature_fraction |101| Random Forest | n_estimators, max_depth, min_samples_split |102| Neural Networks | learning_rate, batch_size, layers, dropout |103104### 5. Model Evaluation105106**Evaluation Best Practices:**107108- Always use held-out test set for final evaluation109- Use cross-validation during development110- Check for overfitting (train vs validation gap)111- Evaluate on multiple metrics112- Analyze errors qualitatively113114**Handling Imbalanced Data:**115116- Resampling: SMOTE, undersampling117- Class weights: weighted loss functions118- Threshold tuning: optimize decision threshold119- Evaluation: use PR-AUC over ROC-AUC120121### 6. Production Deployment122123**Model Serving Patterns:**124125- REST API (Flask, FastAPI, TF Serving)126- Batch inference (scheduled jobs)127- Streaming (real-time predictions)128- Edge deployment (mobile, IoT)129130**Production Considerations:**131132- Latency requirements (p50, p95, p99)133- Throughput (requests per second)134- Model size and memory footprint135- Fallback strategies136- A/B testing framework137138### 7. Monitoring & Maintenance139140**What to Monitor:**141142- Prediction latency143- Input feature distributions (data drift)144- Prediction distributions (concept drift)145- Model performance metrics146- Error rates and types147148**Retraining Triggers:**149150- Performance degradation below threshold151- Significant data drift detected152- Scheduled retraining (daily, weekly)153- New training data available154155## MLOps Best Practices156157### Experiment Tracking158159Track for every experiment:160161- Code version (git commit)162- Data version (hash or version ID)163- Hyperparameters164- Metrics (train, validation, test)165- Model artifacts166- Environment (packages, versions)167168### Model Versioning169170```171models/172├── model_v1.0.0/173│ ├── model.pkl174│ ├── metadata.json175│ ├── requirements.txt176│ └── metrics.json177├── model_v1.1.0/178└── model_v2.0.0/179```180181### CI/CD for ML1821831. **Continuous Integration:**184 - Data validation tests185 - Model training tests186 - Performance regression tests1871882. **Continuous Deployment:**189 - Staging environment validation190 - Shadow mode testing191 - Gradual rollout (canary)192 - Automatic rollback193194## Reference Files195196For detailed patterns and code examples, load reference files as needed:197198- **`references/preprocessing.md`** - Data preprocessing patterns and feature engineering techniques199- **`references/model_patterns.md`** - Model architecture patterns and implementation examples200- **`references/evaluation.md`** - Comprehensive evaluation strategies and metrics201202## Integration with Other Skills203204- **performance** - For optimizing inference latency205- **testing** - For ML-specific testing patterns206- **database-optimization** - For feature store queries207- **debugging** - For model debugging and error analysis
aiskillstore