Data Pipeline

Data Pipeline — data pipeline ·

data pipelineof count→transformation→-based→verification→monitoring inthisbefore teamthis to in ·.

execution

inthisbefore team — 5peoplethis SendMessageas direct and verification.

inthisbefore setup

workflow

Phase 1: (this direct count)

1. user from :
   • **data **: system(DB, API, day, tree)
   • ** system**: data lower, data this, analysis
   • business required: cycle, SLA, data also
   • ** ** (optional): vendor, existing stack, budget
   • existing day (optional): user provided schema, query, configuration day
2. _workspace/ project rootin creation
3. to _workspace/00_input.mdin
4. existing daythis _workspace/in and corresponding Phase cases
5. request scopein execution decision ( " per " )

Phase 2: team setup and execution

team setupand . between of and :

2a()and 2b(scheduling) parallel execution. 1(architecture)inonly ofloweras in startto count .

team between :

• etl-architect completed → quality-managerto schema·business rule before, schedulerto of·resource required before, monitoringto core metric before
• quality-manager completed → schedulerto verification insert location· cases before, monitoringto metric·SLA before
• scheduler completed → monitoringto DAG execution metric before
• reviewer all verification. 🔴 required modification corresponding inthisbeforeto modification request → → verification (maximum 2)

Phase 3: integrated and final

reviewerof report as final :

1. _workspace/ within all day confirmation
2. review reportof 🔴 required modificationthis confirmation
3. final userto report:
   • ETL architecture — 01_etl_architecture.md
   • plan — 02_data_quality_plan.md
   • scheduling configuration — 03_scheduler_config.md
   • monitoring configuration — 04_monitoring_setup.md
   • review report — 05_review_report.md
   • pipeline code — pipeline_code/

per

user requestof scopein inthisbefore :

existing day for: user schema, DAG code etc. existing day providedlower, corresponding day _workspace/of -based locationin and corresponding phaseof inthisbefore cases.

data before as

daypeople : {}_{inthisbefore}_{}.{extension}

error

test

normal

****: "PostgreSQL order data BigQuery data loweras -basedlower dayday pipeline . Airflow as." ** result**:

• ETL architecture: PostgreSQL CDC/minutes as strategy, Raw→Staging→Curated→Analytics 4this
• plan: order data P0/P1/P2 verification rule, revenue aggregation cross-check
• scheduling: Airflow DAG code(extract→stage→curate→analytics→quality_check), retry policy
• monitoring: dayday -based casescount, latencybetween, count dashboard
• review: beforeitem

existing day for

****: "this dbt model dayin about data verificationand monitoring " + dbt model day ** result**:

• existing dbt model _workspace/in
• • monitoring : quality-manager + monitoring + reviewer
• etl-architect, scheduler cases

error

****: "data pipeline , items " ** result**:

• key day-based pipeline templateas in progress
• per placeholder included, per 3phase
• review reportin " information — after necessary" people

inthisbeforeper extension