Profile My skills Saved Collections Edit profile Submit a skill

AI Dataset Engineering

Practical knowledge for building, augmenting, and processing datasets for AI/LLM training and evaluation. Covers data curation (quality, coverage, quantity, acquisition, annotation), data synthesis (rule-based, simulation, AI-powered), instruction data generation, model distillation, and data processing (inspection, deduplication, cleaning, filtering, formatting). Use this skill when: - Curating or sourcing training data - Designing a data annotation pipeline - Generating synthetic data with LLMs - Distilling a smaller model from a larger one - Processing/cleaning datasets for training or evaluation - Estimating dataset size requirements

ebarti Updated
DevOps & Infra

File contents

AI Dataset Engineering

Knowledge from "AI Engineering" by Chip Huyen (Chapter 8). Practical methods for working with training and evaluation datasets.

Quick Start

  1. Check guidelines.md to find which files to load for your task
  2. Load only relevant files (each topic has knowledge.md, rules.md, examples.md)
  3. Apply guidance to your work

Contents

References

Category Purpose
data-curation Data quality, coverage, quantity, acquisition, annotation
data-synthesis Rule-based synthesis, simulation, AI-powered generation, distillation
data-processing Inspection, deduplication, cleaning, filtering, formatting

Workflows

Task Workflow
Curate, synthesize, and process a training dataset workflows/build-training-dataset.md

Guidelines

See guidelines.md for task-based file selection.

ebarti/skills/tree/main/ai-dataset-engineering commit 8b13e47b41

Browse category

Frequently asked questions

npx skillmds@latest add ebarti/ai-dataset-engineering