Etl Pipeline Builder

ETL Pipeline Builder Skill

Builds and manages ETL (Extract, Transform, Load) pipelines for data migration, supporting incremental loads, CDC, and comprehensive monitoring.

Purpose

Enable data pipeline creation for:

• Source-to-target mapping
• Transformation definition
• Incremental load setup
• CDC configuration
• Pipeline monitoring

Capabilities

1. Source-to-Target Mapping

• Define column mappings
• Handle schema differences
• Configure data type conversions
• Manage derived columns

2. Transformation Definition

• Data type transformations
• Value mappings
• Aggregations
• Lookups and enrichments

3. Incremental Load Setup

• Define watermarks
• Configure incremental columns
• Handle deletes
• Manage merge logic

4. CDC Configuration

• Log-based CDC
• Trigger-based CDC
• Timestamp-based CDC
• Full load comparison

5. Error Handling

• Define retry policies
• Configure dead letter queues
• Handle data quality issues
• Implement alerting

6. Pipeline Monitoring

• Track pipeline metrics
• Monitor data volumes
• Alert on failures
• Generate SLA reports

Tool Integrations

Output Schema

{
  "pipelineId": "string",
  "timestamp": "ISO8601",
  "pipeline": {
    "name": "string",
    "source": {},
    "target": {},
    "mappings": [],
    "transformations": [],
    "schedule": "string"
  },
  "artifacts": {
    "dagFile": "string",
    "configFile": "string",
    "sqlFiles": []
  },
  "deployment": {
    "status": "string",
    "url": "string"
  }
}

Integration with Migration Processes

• database-schema-migration: Data movement
• cloud-migration: Cloud data pipelines
• data-format-migration: Format transformation

Related Skills

• data-migration-validator: Validation
• schema-comparator: Schema mapping

Related Agents

• database-migration-orchestrator: Pipeline orchestration
• data-architect-agent: Pipeline design