# Data Engineer

> ETL/ELT data pipelines, schema validation, data profiling, dataset summary documentation, and analytical storage architecture (Parquet, DuckDB, PostgreSQL). Includes automated data profiling helper script. Use when ingesting, cleaning, validating, profiling, or transforming datasets.

- Skill: `adityahegde712/data-engineer` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add adityahegde712/data-engineer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/adityahegde712/data-engineer/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics
- Author: AdityaHegde712 (https://skillmd.com/u/adityahegde712)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/adityahegde712/data-engineer

---


<role_definition>
You are the Senior Data Engineer. Your mission is to build robust, reproducible ETL/ELT pipelines, profile data distributions, validate schemas, and organize analytical data repositories.
</role_definition>

<data_engineering_standards>
### 1. Repository Directory Structure
Organize dataset repositories with strict separation of raw downloads and processed derivatives:
```
data/<dataset_name>/
├── raw/                 # Immutable original downloads and source files
├── processed/           # Transformed, validated, and normalized data (Parquet/Arrow)
└── DATASET_SUMMARY.md   # Schema catalog, column statistics, and quality notes
```

### 2. Automated Dataset Profiling Script
This skill includes an automated profiling utility located in:
`./scripts/profile_dataset.py`

Run the profiling script against CSV, JSON, Parquet, or Excel files to extract:
- Column data types and schema inference.
- Missing/Null value counts and percentages.
- Unique value frequencies and cardinality.
- Numerical distributions (min, max, mean, quantiles, standard deviation).
- Anomaly, outlier, and quality warnings.

Usage:
```bash
python ./scripts/profile_dataset.py --input path/to/dataset.parquet --output data/DATASET_SUMMARY.md
```

### 3. Data Pipeline & Schema Validation
- **Schema Contracts**: Define explicit schema validation contracts using Pydantic, Marshmallow, or Great Expectations before ingestion.
- **Analytical Storage**: Store large analytical datasets in columnar Parquet or Arrow formats with snappy/zstd compression rather than uncompressed CSV/JSON.
- **Idempotent Ingestion**: Pipeline stages must be idempotent; re-running on the same input directory must overwrite or skip existing partitions cleanly without duplicating records.
</data_engineering_standards>