# Dataset Pipeline

> PyTorch Dataset/DataLoader patterns for efficient, reproducible data loading and augmentation.

- Skill: `aselimc/dataset-pipeline` (Agent Skill)
- Install (CLI): `npx skillmds@latest add aselimc/dataset-pipeline`
- Raw SKILL.md: https://api.skillmd.com/api/skills/aselimc/dataset-pipeline/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: aselimc (https://skillmd.com/u/aselimc)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/aselimc/dataset-pipeline

---


# Dataset Pipeline

## Standard Pattern
```python
class MyDataset(Dataset):
    def __init__(self, root, split, transform=None):
        self.samples = self._load_manifest(root, split)
        self.transform = transform
    def __len__(self): return len(self.samples)
    def __getitem__(self, idx):
        img, label = self._load_sample(self.samples[idx])
        if self.transform: img = self.transform(img)
        return img, label
```

## DataLoader Config
```python
DataLoader(dataset, batch_size=32, shuffle=True, num_workers=8,
           pin_memory=True, prefetch_factor=2, persistent_workers=True)
```

## Augmentation
- **CV**: albumentations or torchvision.transforms.v2
- **NLP**: tokenizer pipelines (HuggingFace tokenizers)
- **Audio**: torchaudio.transforms

## Rules
- Reproducible splits: seeded RNG, save split indices
- Validate data at load time (check shapes, dtypes, NaN)
- Use memory-mapped formats for large datasets (webdataset, HF datasets)
- Never augment validation/test data (except TTA at inference)

## Key Libraries
torch.utils.data, albumentations, torchvision.transforms, HuggingFace datasets, webdataset