Data Engineering
Operating contract
- Identify the requested deliverable, then select exactly one atomic task below.
- Read that task file completely before acting.
- Read references/lifecycle-standard.md and use the task's profile, risk tier and execution path.
- Load only the company context and adapter references needed for the selected task.
- On a cross-role handoff, compare logical ID and SHA-256 in
references/shared-reference-manifest.json; reuse an already loaded shared reference when both match instead of loading its duplicate.
- Inspect real artifacts and systems before making change-sensitive claims.
- Apply Definition of Ready, stage gates, test strategy, Definition of Done, approval and handoff requirements.
- Do not invent access, approvals, successful execution, test results or business confirmation.
- For any Git-backed mutation, regardless of the task's verb or profile, require a pre-change success/scope contract, a post-change
core-audit-change-scope, and fresh core-verify-deliverable evidence before completion.
- For learning, coaching or skill-transition work, resolve the learner-memory interoperability contract. Reuse only relevant verified summaries; never infer mastery from exposure, and expand stale, uncertain, changed-version or safety-critical prerequisites.
Performance routing
- Diagnose an existing SQL or Spark plan and form a falsifiable bottleneck hypothesis ->
de-analyze-execution-plan.
- Implement and benchmark a controlled pipeline performance change after diagnosis ->
de-optimize-pipeline-performance.
Never prescribe an optimization from syntax alone when an engine-native execution plan and workload metrics can be inspected.
Stack-native adapter routing
After selecting the task, detect the real stack/version and read only the matching adapter below plus the adapter index. Do not load every adapter.
- airflow
- dbt
- spark
- kafka-flink
- snowflake
- bigquery
- databricks
- microsoft-fabric
Atomic task routing
- Plan, define, design, map, specify or create a proposed artifact (3 tasks): read references/catalog-plan-design.md.
- Build, implement, configure, teach, interview or deliver an artifact (11 tasks): read references/catalog-build-deliver.md.
- Inspect, analyze, test, review, validate, assess, certify or audit (4 tasks): read references/catalog-test-assure.md.
- Deploy, release, monitor, recover, migrate, optimize, retire or improve (7 tasks): read references/catalog-operate-improve.md.
Read only the best-matching catalog. If intent remains ambiguous, inspect a second catalog; do not load all catalogs by default. Select one task by primary deliverable, then read its contract completely.
Completion response
State the selected task ID, primary deliverable, evidence inspected, validation performed, approval status, residual risks and next owner. A draft or plan is not an executed production outcome.
1---2name: data-engineering3description: Design, build, test, diagnose execution plans and operate batch, API, file, CDC and streaming pipelines with idempotency, schema evolution, reconciliation, recovery and runbooks. Use for Data Engineer ingestion, performance or pipeline work. Route feature pipelines and model serving to machine-learning-engineering, dbt-style modelling to analytics-engineering, and catalog or lineage harvesting to metadata-engineering-and-catalog.4---56# Data Engineering78## Operating contract9101. Identify the requested deliverable, then select exactly one atomic task below.112. Read that task file completely before acting.123. Read [references/lifecycle-standard.md](references/lifecycle-standard.md) and use the task's profile, risk tier and execution path.134. Load only the company context and adapter references needed for the selected task.145. On a cross-role handoff, compare logical ID and SHA-256 in `references/shared-reference-manifest.json`; reuse an already loaded shared reference when both match instead of loading its duplicate.156. Inspect real artifacts and systems before making change-sensitive claims.167. Apply Definition of Ready, stage gates, test strategy, Definition of Done, approval and handoff requirements.178. Do not invent access, approvals, successful execution, test results or business confirmation.189. For any Git-backed mutation, regardless of the task's verb or profile, require a pre-change success/scope contract, a post-change `core-audit-change-scope`, and fresh `core-verify-deliverable` evidence before completion.1910. For learning, coaching or skill-transition work, resolve [the learner-memory interoperability contract](references/learning-memory-interoperability.md). Reuse only relevant verified summaries; never infer mastery from exposure, and expand stale, uncertain, changed-version or safety-critical prerequisites.2021## Performance routing2223- Diagnose an existing SQL or Spark plan and form a falsifiable bottleneck hypothesis -> `de-analyze-execution-plan`.24- Implement and benchmark a controlled pipeline performance change after diagnosis -> `de-optimize-pipeline-performance`.2526Never prescribe an optimization from syntax alone when an engine-native execution plan and workload metrics can be inspected.272829## Stack-native adapter routing3031After selecting the task, detect the real stack/version and read only the matching adapter below plus [the adapter index](references/technology-adapters.md). Do not load every adapter.3233- [airflow](references/adapter-airflow.md)34- [dbt](references/adapter-dbt.md)35- [spark](references/adapter-spark.md)36- [kafka-flink](references/adapter-kafka-flink.md)37- [snowflake](references/adapter-snowflake.md)38- [bigquery](references/adapter-bigquery.md)39- [databricks](references/adapter-databricks.md)40- [microsoft-fabric](references/adapter-microsoft-fabric.md)4142## Atomic task routing4344- **Plan, define, design, map, specify or create a proposed artifact** (3 tasks): read [references/catalog-plan-design.md](references/catalog-plan-design.md).45- **Build, implement, configure, teach, interview or deliver an artifact** (11 tasks): read [references/catalog-build-deliver.md](references/catalog-build-deliver.md).46- **Inspect, analyze, test, review, validate, assess, certify or audit** (4 tasks): read [references/catalog-test-assure.md](references/catalog-test-assure.md).47- **Deploy, release, monitor, recover, migrate, optimize, retire or improve** (7 tasks): read [references/catalog-operate-improve.md](references/catalog-operate-improve.md).4849Read only the best-matching catalog. If intent remains ambiguous, inspect a second catalog; do not load all catalogs by default. Select one task by primary deliverable, then read its contract completely.5051## Completion response5253State the selected task ID, primary deliverable, evidence inspected, validation performed, approval status, residual risks and next owner. A draft or plan is not an executed production outcome.
kina2711