# AI Dataset Engineering

> Practical knowledge for building, augmenting, and processing datasets for AI/LLM training and evaluation. Covers data curation (quality, coverage, quantity, acquisition, annotation), data synthesis (rule-based, simulation, AI-powered), instruction data generation, model distillation, and data processing (inspection, deduplication, cleaning, filtering, formatting). Use this skill when: - Curating or sourcing training data - Designing a data annotation pipeline - Generating synthetic data with LLMs - Distilling a smaller model from a larger one - Processing/cleaning datasets for training or evaluation - Estimating dataset size requirements

- Skill: `ebarti/ai-dataset-engineering` (Agent Skill, multi-file: 15 files)
- Install (CLI): `npx skillmds@latest add ebarti/ai-dataset-engineering`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ebarti/ai-dataset-engineering/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: ebarti (https://skillmd.com/u/ebarti)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ebarti/ai-dataset-engineering

---


# AI Dataset Engineering

Knowledge from "AI Engineering" by Chip Huyen (Chapter 8). Practical methods for working with training and evaluation datasets.

## Quick Start

1. Check `guidelines.md` to find which files to load for your task
2. Load only relevant files (each topic has knowledge.md, rules.md, examples.md)
3. Apply guidance to your work

## Contents

### References

| Category | Purpose |
|----------|---------|
| `data-curation` | Data quality, coverage, quantity, acquisition, annotation |
| `data-synthesis` | Rule-based synthesis, simulation, AI-powered generation, distillation |
| `data-processing` | Inspection, deduplication, cleaning, filtering, formatting |

### Workflows

| Task | Workflow |
|------|----------|
| Curate, synthesize, and process a training dataset | `workflows/build-training-dataset.md` |

## Guidelines

See `guidelines.md` for task-based file selection.