Python Data Engineering And Pipeline Packaging
Overview
Use this skill when Python is the main implementation language for data pipelines or operational data tooling. It helps agents structure jobs as maintainable packages instead of loose scripts, choose the right execution boundary, manage dependencies explicitly, and keep runtime behavior testable and production-safe.
When to Use
- building or modifying
Python data pipelines
- packaging
PySpark, ingestion, validation, or orchestration helper code
- moving from notebooks or scripts into production-ready modules
- managing dependency, environment, and runtime issues in
Python
- adding CLI entry points, test harnesses, or local development workflows
Do not treat a working script as a production design just because it runs once.
Workflow
Define the role of the Python code.
Clarify whether it is:
- a single-node transform
- a
PySpark job entry point
- an orchestration helper
- a validation or reconciliation tool
- an integration or extraction service
Package logic into explicit modules.
Prefer:
- versioned packages
- reusable modules
- clear CLI or job entry points
- isolated configuration
- minimal hidden global state
Make dependency management real.
Define:
- environment model
- pinned dependency strategy
- native or system dependency assumptions
- compatibility with runtime platforms such as
Airflow, Spark, or container images
Keep runtime boundaries explicit.
Decide:
- what runs locally versus distributed
- what belongs in orchestration versus the job package
- how configuration, secrets, and environment values are supplied
- how logs, retries, and exits behave operationally
Prove the package is maintainable.
Require:
- targeted tests
- representative input cases
- type or interface clarity where useful
- reproducible local execution
Common Rationalizations
| Rationalization |
Reality |
| "It is only a small Python script." |
Small scripts often become critical pipeline entry points with no packaging or test discipline. |
| "We can keep the business logic in the DAG or notebook." |
Hidden logic in orchestration or notebook state becomes hard to test, reuse, and debug. |
| "Requirements are enough documentation." |
Dependency files do not explain runtime assumptions, entry points, or platform compatibility. |
Red Flags
- pipeline logic lives mostly in one script with no reusable module structure
- notebooks, DAGs, and job code duplicate the same transformation logic
- dependency versions are implicit or environment-specific
- local runs and deployed runs behave differently with no explanation
- secrets or environment assumptions are embedded in code
Verification
1---2name: python-data-engineering-and-pipeline-packaging3description: Guides agents through Python-based data engineering implementation. Use when building or modifying Python ingestion jobs, orchestration helpers, PySpark entry points, validation code, packaging, dependency management, or operational CLI workflows.4---56# Python Data Engineering And Pipeline Packaging78## Overview910Use this skill when `Python` is the main implementation language for data pipelines or operational data tooling. It helps agents structure jobs as maintainable packages instead of loose scripts, choose the right execution boundary, manage dependencies explicitly, and keep runtime behavior testable and production-safe.1112## When to Use1314- building or modifying `Python` data pipelines15- packaging `PySpark`, ingestion, validation, or orchestration helper code16- moving from notebooks or scripts into production-ready modules17- managing dependency, environment, and runtime issues in `Python`18- adding CLI entry points, test harnesses, or local development workflows1920Do not treat a working script as a production design just because it runs once.2122## Workflow23241. Define the role of the Python code.25 Clarify whether it is:26 - a single-node transform27 - a `PySpark` job entry point28 - an orchestration helper29 - a validation or reconciliation tool30 - an integration or extraction service31322. Package logic into explicit modules.33 Prefer:34 - versioned packages35 - reusable modules36 - clear CLI or job entry points37 - isolated configuration38 - minimal hidden global state39403. Make dependency management real.41 Define:42 - environment model43 - pinned dependency strategy44 - native or system dependency assumptions45 - compatibility with runtime platforms such as `Airflow`, `Spark`, or container images46474. Keep runtime boundaries explicit.48 Decide:49 - what runs locally versus distributed50 - what belongs in orchestration versus the job package51 - how configuration, secrets, and environment values are supplied52 - how logs, retries, and exits behave operationally53545. Prove the package is maintainable.55 Require:56 - targeted tests57 - representative input cases58 - type or interface clarity where useful59 - reproducible local execution6061## Common Rationalizations6263| Rationalization | Reality |64| --- | --- |65| "It is only a small Python script." | Small scripts often become critical pipeline entry points with no packaging or test discipline. |66| "We can keep the business logic in the DAG or notebook." | Hidden logic in orchestration or notebook state becomes hard to test, reuse, and debug. |67| "Requirements are enough documentation." | Dependency files do not explain runtime assumptions, entry points, or platform compatibility. |6869## Red Flags7071- pipeline logic lives mostly in one script with no reusable module structure72- notebooks, DAGs, and job code duplicate the same transformation logic73- dependency versions are implicit or environment-specific74- local runs and deployed runs behave differently with no explanation75- secrets or environment assumptions are embedded in code7677## Verification7879- [ ] The Python code has a clear package and entry-point shape80- [ ] Runtime, dependency, and environment assumptions are explicit81- [ ] Orchestration code and business logic are separated82- [ ] Tests or reproducible execution paths exist for the important logic
vaquarkhan