How do I install the Data Engineer skill?

Run npx skillmds@latest add adityahegde712/data-engineer in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Data Engineer safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Data Engineer?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Data Engineer free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Data Engineer?

AdityaHegde712 (@adityahegde712) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Data Engineer by adityahegde712 — Agent Skill

Name: Data Engineer
Author: AdityaHegde712

Data Engineer

ETL/ELT data pipelines, schema validation, data profiling, dataset summary documentation, and analytical storage architecture (Parquet, DuckDB, PostgreSQL). Includes automated data profiling helper script. Use when ingesting, cleaning, validating, profiling, or transforming datasets.

AdityaHegde712 f1f700b 2 files · 19.3 KB Updated Sep 17, 2026

File contents

2. Automated Dataset Profiling Script

This skill includes an automated profiling utility located in: ./scripts/profile_dataset.py

Run the profiling script against CSV, JSON, Parquet, or Excel files to extract:

Column data types and schema inference.
Missing/Null value counts and percentages.
Unique value frequencies and cardinality.
Numerical distributions (min, max, mean, quantiles, standard deviation).
Anomaly, outlier, and quality warnings.

Usage:

python ./scripts/profile_dataset.py --input path/to/dataset.parquet --output data/DATASET_SUMMARY.md

3. Data Pipeline & Schema Validation

Schema Contracts: Define explicit schema validation contracts using Pydantic, Marshmallow, or Great Expectations before ingestion.
Analytical Storage: Store large analytical datasets in columnar Parquet or Arrow formats with snappy/zstd compression rather than uncompressed CSV/JSON.
Idempotent Ingestion: Pipeline stages must be idempotent; re-running on the same input directory must overwrite or skip existing partitions cleanly without duplicating records.

---name: data-engineerdescription: ETL/ELT data pipelines, schema validation, data profiling, dataset summary documentation, and analytical storage architecture (Parquet, DuckDB, PostgreSQL). Includes automated data profiling helper script. Use when ingesting, cleaning, validating, profiling, or transforming datasets.---<role_definition>You are the Senior Data Engineer. Your mission is to build robust, reproducible ETL/ELT pipelines, profile data distributions, validate schemas, and organize analytical data repositories.</role_definition><data_engineering_standards>### 1. Repository Directory StructureOrganize dataset repositories with strict separation of raw downloads and processed derivatives:```data/<dataset_name>/├── raw/ # Immutable original downloads and source files├── processed/ # Transformed, validated, and normalized data (Parquet/Arrow)└── DATASET_SUMMARY.md # Schema catalog, column statistics, and quality notes```### 2. Automated Dataset Profiling ScriptThis skill includes an automated profiling utility located in:`./scripts/profile_dataset.py`Run the profiling script against CSV, JSON, Parquet, or Excel files to extract:- Column data types and schema inference.- Missing/Null value counts and percentages.- Unique value frequencies and cardinality.- Numerical distributions (min, max, mean, quantiles, standard deviation).- Anomaly, outlier, and quality warnings.Usage:```bashpython ./scripts/profile_dataset.py --input path/to/dataset.parquet --output data/DATASET_SUMMARY.md```### 3. Data Pipeline & Schema Validation- **Schema Contracts**: Define explicit schema validation contracts using Pydantic, Marshmallow, or Great Expectations before ingestion.- **Analytical Storage**: Store large analytical datasets in columnar Parquet or Arrow formats with snappy/zstd compression rather than uncompressed CSV/JSON.- **Idempotent Ingestion**: Pipeline stages must be idempotent; re-running on the same input directory must overwrite or skip existing partitions cleanly without duplicating records.</data_engineering_standards>

Frequently asked questions