# Data Pipeline Engineering

> ETL workflow design, automated data fetching, version tracking, and pipeline orchestration expertise

- Skill: `hack23/data-pipeline-engineering` (Agent Skill)
- Install (CLI): `npx skillmds@latest add hack23/data-pipeline-engineering`
- Raw SKILL.md: https://api.skillmd.com/api/skills/hack23/data-pipeline-engineering/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- License: CC-BY-4.0
- Author: Hack23 (https://skillmd.com/u/hack23)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/hack23/data-pipeline-engineering

---


# Data Pipeline Engineering Skill


## 🔴 AI FIRST Quality Principle

> **Apply the AI FIRST principle: never accept first-pass quality. Minimum 2 iterations. Read all output, improve every section. No shortcuts.**

## Purpose
Expert knowledge in designing robust ETL (Extract, Transform, Load) pipelines for automated data processing, focusing on reliability, monitoring, and maintainability.

## Core Principles
1. **Idempotency** - Pipeline runs produce same results
2. **Observability** - Full visibility into pipeline health
3. **Error Recovery** - Graceful handling of failures
4. **Version Tracking** - Track all data changes
5. **Monitoring** - Real-time pipeline health checks

## Enforces
- ETL workflow patterns (Extract → Transform → Load)
- Automated scheduling (cron, GitHub Actions)
- Data versioning and archival
- Pipeline health monitoring
- Error recovery strategies
- Audit logging

## When to Use
- Building automated data pipelines
- Scheduling data fetching workflows
- Implementing data versioning
- Monitoring pipeline health
- Designing error recovery

## References
- [GitHub Actions](https://docs.github.com/en/actions)
- [ETL Best Practices](https://en.wikipedia.org/wiki/Extract,_transform,_load)

---
**Version**: 1.0 | **Last Updated**: 2026-02-06 | **Category**: Development & Operations