Tracing Upstream Lineage

Upstream Lineage: Sources

Trace the origins of data and answer "Where does this data come from?"

Lineage Investigation

Step 1: Identify the Target Type

Determine what we are tracing:

• Table
• Column
• DAG

Step 2: Find the Producing DAG

1. List DAGs: use list_active_dags and list_paused_dags
2. Read DAG source: use get_dag_source_code
3. If a run exists, use analyse_dag_latest_run to see tasks and logs

Step 3: Trace Data Sources

From the DAG code, identify source tables and systems:

• SQL sources in FROM or JOIN clauses
• External sources via operator hooks or connection IDs
• Files in object storage

Use go_to_connections_view to inspect connection metadata.

Step 4: Build the Lineage Chain

Example:

TARGET: analytics.orders_daily
    ^
    +-- DAG: etl_daily_orders
            ^
            +-- SOURCE: raw.orders
            |
            +-- SOURCE: dim.customers

Step 5: Check Source Health

• Use get_dag_runs or get_dag_history on upstream DAGs
• For logs, use go_to_dag_log_view

Lineage for Columns

1. Find the column in the target table schema
2. Search DAG source for references
3. Trace transformations and mappings

Output: Lineage Report

Include:

• Summary of sources
• Lineage diagram
• Source details (connections, freshness)
• Transformation chain
• Data quality implications

Related Skills

• checking-freshness
• debugging-dags
• tracing-downstream-lineage
• annotating-task-lineage
• creating-openlineage-extractors