ML Experiment Skill
Description
Design, implement, and evaluate machine learning experiments with reproducible workflows, proper baselines, and statistical analysis.
Tools Used
jupyter_execute - Execute ML code in Python (auto-switches to Jupyter)jupyter_notebook - Manage experiment notebooksupdate_notebook - Set up experiment cellsupdate_latex - Write experiment results to paperslatex_compile - Compile CS conference papers (auto-switches to LaTeX)arxiv_to_prompt - Read related work from arXiv papersupdate_notes - Write experiment logs and analysis summaries
Capabilities
Experiment Design
- Proper train/validation/test splits
- Cross-validation and bootstrap confidence intervals
- Ablation study design
- Hyperparameter search (grid, random, Bayesian)
Implementation
- PyTorch and TensorFlow model building
- Data loading and augmentation pipelines
- Training loops with logging and checkpointing
- Distributed training setup
Evaluation
- Standard metrics per task (accuracy, F1, BLEU, mAP, etc.)
- Statistical significance testing (paired t-test, bootstrap)
- Comparison with baselines
- Error analysis and visualization
Usage Patterns
Run an Experiment
When user says: "Train a model for [task]"
- Clarify dataset, metrics, and baselines
- Implement data loading and preprocessing
- Build model architecture
- Train with proper logging
- Evaluate and compare to baselines
- Report results with confidence intervals
Reproduce a Paper
When user says: "Reproduce [paper title/arXiv ID]"
- Fetch paper using arxiv_to_prompt
- Extract key method details
- Implement core algorithm
- Run experiments matching paper setup
- Compare results to reported numbers
Tool Examples
Train and evaluate a classifier
# via jupyter_execute
import torch
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# ... train model ...
print(classification_report(y_test, predictions))
Run ablation study
# via jupyter_execute
configs = [
{"name": "full", "use_augmentation": True, "use_dropout": True},
{"name": "no_aug", "use_augmentation": False, "use_dropout": True},
{"name": "no_dropout", "use_augmentation": True, "use_dropout": False},
]
results = {c["name"]: train_and_eval(**c) for c in configs}
Validation checkpoints
- Verify data shapes match expected dimensions before training
- Check that loss is decreasing after the first few epochs
- Confirm test set has no overlap with training data
1---2name: ml-experiment3description: Design and run machine learning experiments with proper evaluation using jupyter_execute, including training, benchmarking, and ablation studies. Use when the user wants to train models, compare algorithms, run ablation studies, evaluate ML performance, or reproduce paper results.4---5
6# ML Experiment Skill
7
8## Description
9Design, implement, and evaluate machine learning experiments with reproducible workflows, proper baselines, and statistical analysis.
10
11## Tools Used
12- `jupyter_execute` - Execute ML code in Python (auto-switches to Jupyter)
13- `jupyter_notebook` - Manage experiment notebooks
14- `update_notebook` - Set up experiment cells
15- `update_latex` - Write experiment results to papers
16- `latex_compile` - Compile CS conference papers (auto-switches to LaTeX)
17- `arxiv_to_prompt` - Read related work from arXiv papers
18- `update_notes` - Write experiment logs and analysis summaries
19
20## Capabilities
21
22### Experiment Design
23- Proper train/validation/test splits
24- Cross-validation and bootstrap confidence intervals
25- Ablation study design
26- Hyperparameter search (grid, random, Bayesian)
27
28### Implementation
29- PyTorch and TensorFlow model building
30- Data loading and augmentation pipelines
31- Training loops with logging and checkpointing
32- Distributed training setup
33
34### Evaluation
35- Standard metrics per task (accuracy, F1, BLEU, mAP, etc.)
36- Statistical significance testing (paired t-test, bootstrap)
37- Comparison with baselines
38- Error analysis and visualization
39
40## Usage Patterns
41
42### Run an Experiment
43When user says: "Train a model for [task]"
441. Clarify dataset, metrics, and baselines
452. Implement data loading and preprocessing
463. Build model architecture
474. Train with proper logging
485. Evaluate and compare to baselines
496. Report results with confidence intervals
50
51### Reproduce a Paper
52When user says: "Reproduce [paper title/arXiv ID]"
531. Fetch paper using arxiv_to_prompt
542. Extract key method details
553. Implement core algorithm
564. Run experiments matching paper setup
575. Compare results to reported numbers
58
59## Tool Examples
60
61### Train and evaluate a classifier
62```python
63# via jupyter_execute
64import torch
65from sklearn.model_selection import train_test_split
66from sklearn.metrics import classification_report
67
68X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
69# ... train model ...
70print(classification_report(y_test, predictions))
71```
72
73### Run ablation study
74```python
75# via jupyter_execute
76configs = [
77 {"name": "full", "use_augmentation": True, "use_dropout": True},
78 {"name": "no_aug", "use_augmentation": False, "use_dropout": True},
79 {"name": "no_dropout", "use_augmentation": True, "use_dropout": False},
80]
81results = {c["name"]: train_and_eval(**c) for c in configs}
82```
83
84### Validation checkpoints
85- Verify data shapes match expected dimensions before training
86- Check that loss is decreasing after the first few epochs
87- Confirm test set has no overlap with training data
gabrielmoreira