Prepare Dataset
Load, preprocess, and validate datasets for machine learning model training including normalization and augmentation.
When to Use
- Setting up data pipelines for training
- Normalizing and cleaning raw data
- Splitting into train/validation/test sets
- Applying data augmentation
Quick Reference
# Dataset preparation pipeline
class DatasetLoader:
def load(self, path: str) -> Tuple[ndarray, ndarray]:
# Load raw data
pass
def normalize(self, data: ndarray) -> ndarray:
# Normalize to [0, 1] or standardize
pass
def split(self, data: ndarray, ratios: Tuple[float, float, float]):
# Split into train/val/test
pass
def augment(self, data: ndarray) -> ndarray:
# Apply transformations if needed
pass
Workflow
- Load raw data: Read dataset from file (CSV, HDF5, NumPy)
- Validate data: Check shape, dtype, missing values
- Preprocess: Normalize, standardize, encode categorical features
- Split sets: Create train/validation/test splits
- Augment data: Apply transformations if needed (rotation, flip, etc.)
Output Format
Dataset preparation report:
- Raw data shape and statistics
- Data validation results (missing values, outliers)
- Preprocessing applied (normalization, encoding)
- Train/val/test split sizes
- Final dataset shape and statistics
- Augmentation transformations applied
References
- See
extract-hyperparameters skill for data preprocessing config
- See
evaluate-model skill for test set evaluation
- See
/notes/review/mojo-ml-patterns.md for Mojo data loading
1---2name: prepare-dataset3description: Process and validate datasets for training. Use when setting up data pipelines.4---5
6# Prepare Dataset
7
8Load, preprocess, and validate datasets for machine learning model training including normalization and augmentation.
9
10## When to Use
11
12- Setting up data pipelines for training
13- Normalizing and cleaning raw data
14- Splitting into train/validation/test sets
15- Applying data augmentation
16
17## Quick Reference
18
19```python
20# Dataset preparation pipeline
21class DatasetLoader:
22 def load(self, path: str) -> Tuple[ndarray, ndarray]:
23 # Load raw data
24 pass
25
26 def normalize(self, data: ndarray) -> ndarray:
27 # Normalize to [0, 1] or standardize
28 pass
29
30 def split(self, data: ndarray, ratios: Tuple[float, float, float]):
31 # Split into train/val/test
32 pass
33
34 def augment(self, data: ndarray) -> ndarray:
35 # Apply transformations if needed
36 pass
37```
38
39## Workflow
40
411. **Load raw data**: Read dataset from file (CSV, HDF5, NumPy)
422. **Validate data**: Check shape, dtype, missing values
433. **Preprocess**: Normalize, standardize, encode categorical features
444. **Split sets**: Create train/validation/test splits
455. **Augment data**: Apply transformations if needed (rotation, flip, etc.)
46
47## Output Format
48
49Dataset preparation report:
50
51- Raw data shape and statistics
52- Data validation results (missing values, outliers)
53- Preprocessing applied (normalization, encoding)
54- Train/val/test split sizes
55- Final dataset shape and statistics
56- Augmentation transformations applied
57
58## References
59
60- See `extract-hyperparameters` skill for data preprocessing config
61- See `evaluate-model` skill for test set evaluation
62- See `/notes/review/mojo-ml-patterns.md` for Mojo data loading
majiayu000