ETL ELT And Modernization Strategy
Overview
Use this skill when the hard part is not a single job, but deciding where transformations should run and how a data estate should modernize over time. It helps agents reason about ETL versus ELT, pushdown versus external compute, orchestration boundaries, migration sequencing, and proof of parity during modernization.
When to Use
- choosing between
ETL, ELT, or hybrid transformation patterns
- moving from legacy ETL tools into warehouse, dbt, Spark, or lakehouse execution
- redesigning ingestion and transformation boundaries across raw, curated, and publish layers
- reducing operational sprawl caused by duplicate transformation logic
- modernizing batch-first estates without breaking existing delivery expectations
Do not assume ELT is always better just because the warehouse is powerful.
Workflow
Define the transformation problem clearly.
Clarify:
- source latency and volume
- data quality expectations
- transformation complexity
- cost sensitivity
- publish or consumption latency
Map the current execution estate.
Include:
- where extraction happens
- where transformations happen today
- what logic is duplicated across tools
- where lineage or observability breaks
- what jobs are hardest to change safely
Choose the right execution boundary.
Consider:
ETL when data must be reshaped or protected before landingELT when warehouse or lakehouse pushdown improves maintainability and scaling- hybrid patterns when extraction, privacy controls, or heavy preprocessing must happen before durable load
Plan the modernization path.
Decide:
- what stays temporarily on the old path
- what moves first
- how parity will be measured
- how cutover and rollback will work
Prove the new shape operationally.
Require:
- reconciliation evidence
- cost and performance review
- lineage continuity
- ownership and support readiness
Common Rationalizations
| Rationalization |
Reality |
| "Everything should become ELT." |
Some workloads still need pre-load shaping, masking, or protocol-specific extraction controls. |
| "The ETL tool is the problem." |
The real issue may be unclear ownership, poor contracts, or duplicated logic across layers. |
| "We can rewrite all transformations at once." |
Big-bang modernization usually breaks parity, runbooks, and downstream trust. |
Red Flags
- the same business logic exists in extraction jobs, Spark, and warehouse SQL
- ETL versus ELT is chosen by tool preference instead of workload needs
- modernization plans skip parity, cutover, or rollback
- sensitive fields are moved into ELT layers without revisiting controls
Verification
1---2name: etl-elt-and-modernization-strategy3description: Guides agents through ETL, ELT, and transformation-modernization decisions. Use when choosing execution boundaries, redesigning transformation layers, or moving from legacy ETL estates to warehouse- or lakehouse-centered ELT patterns.4---56# ETL ELT And Modernization Strategy78## Overview910Use this skill when the hard part is not a single job, but deciding where transformations should run and how a data estate should modernize over time. It helps agents reason about `ETL` versus `ELT`, pushdown versus external compute, orchestration boundaries, migration sequencing, and proof of parity during modernization.1112## When to Use1314- choosing between `ETL`, `ELT`, or hybrid transformation patterns15- moving from legacy ETL tools into warehouse, dbt, Spark, or lakehouse execution16- redesigning ingestion and transformation boundaries across raw, curated, and publish layers17- reducing operational sprawl caused by duplicate transformation logic18- modernizing batch-first estates without breaking existing delivery expectations1920Do not assume `ELT` is always better just because the warehouse is powerful.2122## Workflow23241. Define the transformation problem clearly.25 Clarify:26 - source latency and volume27 - data quality expectations28 - transformation complexity29 - cost sensitivity30 - publish or consumption latency31322. Map the current execution estate.33 Include:34 - where extraction happens35 - where transformations happen today36 - what logic is duplicated across tools37 - where lineage or observability breaks38 - what jobs are hardest to change safely39403. Choose the right execution boundary.41 Consider:42 - `ETL` when data must be reshaped or protected before landing43 - `ELT` when warehouse or lakehouse pushdown improves maintainability and scaling44 - hybrid patterns when extraction, privacy controls, or heavy preprocessing must happen before durable load45464. Plan the modernization path.47 Decide:48 - what stays temporarily on the old path49 - what moves first50 - how parity will be measured51 - how cutover and rollback will work52535. Prove the new shape operationally.54 Require:55 - reconciliation evidence56 - cost and performance review57 - lineage continuity58 - ownership and support readiness5960## Common Rationalizations6162| Rationalization | Reality |63| --- | --- |64| "Everything should become ELT." | Some workloads still need pre-load shaping, masking, or protocol-specific extraction controls. |65| "The ETL tool is the problem." | The real issue may be unclear ownership, poor contracts, or duplicated logic across layers. |66| "We can rewrite all transformations at once." | Big-bang modernization usually breaks parity, runbooks, and downstream trust. |6768## Red Flags6970- the same business logic exists in extraction jobs, Spark, and warehouse SQL71- ETL versus ELT is chosen by tool preference instead of workload needs72- modernization plans skip parity, cutover, or rollback73- sensitive fields are moved into ELT layers without revisiting controls7475## Verification7677- [ ] The transformation boundary matches the real workload constraints78- [ ] ETL, ELT, and hybrid choices are explicit rather than assumed79- [ ] Modernization sequencing, parity proof, and rollback are defined80- [ ] Cost, lineage, controls, and support ownership were considered together
vaquarkhan