Lineage

Data Lineage Guide

Overview

Use this skill any time the user wants to understand data flow, trace data origins, identify downstream impacts, or analyze how data pipelines have changed over time. This skill guides users through exploring data lineage relationships in a structured way, ensuring they understand the scope and depth of lineage traversal before executing queries.

Phase 0: Intent Detection and Entry Path Selection

Always start by understanding what the user wants to accomplish and which entry path is most appropriate:

Entry Path A: Direct Lineage Search (Lineage-First)

Use when the user mentions an asset name but doesn't specify a catalog/project context, or when they want to search across all lineage assets.

• Tool: search_lineage_assets
• Best for: "Find the lineage for CUSTOMER_360", "What feeds into the sales table?"
• Supports: Optional filters for technology type and asset type

Entry Path B: Catalog Asset Lookup (Catalog-First)

Use when the user references a known catalog or project asset, or provides specific container context.

• Tools: search_asset → convert_asset_to_lineage_id
• Best for: "Show lineage for the ORDERS table in the sales catalog", "Trace the customer_data asset in AgentTest project"
• Requires: Container context (catalog or project name)

Decision Logic:

• If user mentions "catalog" or "project" name → Use Path B (Catalog-First)
• If user only provides asset name without container → Use Path A (Lineage-First)
• If uncertain → Ask user to clarify or default to Path A

Phase 1: Asset Identification

Locate the starting asset(s) in the lineage graph using the appropriate entry path.

Path A: Direct Lineage Search

Path B: Catalog Asset Lookup

Important Validation:

• Lineage IDs must be exactly 64 hexadecimal characters
• If you receive a shorter ID or UUID, you MUST use convert_asset_to_lineage_id to convert it
• Never proceed to Phase 2 without a valid 64-character lineage ID

Phase 2: Lineage Graph Traversal

Retrieve and present the upstream and downstream lineage relationships.

Critical: Inform User About Traversal Depth

BEFORE calling get_lineage_graph, you MUST:

Executing Lineage Graph Query

Presenting Lineage Results

Special Cases

Finding Path Between Two Assets:

• If user asks "trace from Asset A to Asset B" or "path between X and Y":
  1. Get lineage IDs for both assets (Phase 1)
  2. Call get_lineage_graph with:
     • lineage_ids: [lineage_id_A, lineage_id_B]
     • hop_up: "50"
     • hop_down: "50"
     • ultimate: "" (empty string)
  3. Search the returned graph for the path connecting the two assets

Impact Analysis:

• If user asks "what would break if we changed X?":
  1. Focus on downstream traversal
  2. Set hop_down to "50" for complete impact view
  3. Highlight all dependent tables, reports, and data products

Source Tracing:

• If user asks "where does this data come from originally?":
  1. Focus on upstream traversal
  2. Set hop_up to "50" or use ultimate: "source"
  3. Identify the terminal source asset(s)

Phase 3: Historical Lineage Comparison (Optional)

Note: This phase depends on the list_lineage_versions tool which provides version timestamps for historical comparison.

Use this phase when the user wants to understand how the data pipeline has changed over time.

Natural Language Date Conversion:

• "since Friday" → Calculate the date of the most recent Friday and convert to ISO 8601
• "since last week" → Calculate 7 days ago from current date
• "since last month" → Calculate 30 days ago or first day of previous month
• "in the last 3 days" → Calculate 3 days ago from current date
• "this year" → Use current year start (e.g., "2026-01-01T00:00:00Z")
• Use the current time to calculate relative dates accurately

If user is comparing assets:

• Call get_lineage_comparison with:
  • compared_lineage_assets: Asset IDs to compare
  • base_version: Later date (more recent version)
  • compared_version: Earlier date (older version)

If user is comparing graphs:

• Call get_lineage_comparison with:
  • initial_lineage_assets: Asset IDs that were passed to get_lineage_graph as lineage_ids
  • compared_lineage_assets: All asset IDs returned by get_lineage_graph
  • base_version: Later date (more recent version)
  • compared_version: Earlier date (older version)

Historical Comparison Example Workflow

User: "Has anything changed in the pipeline for our revenue dashboard in the last month?"

1. Find the revenue dashboard asset (Phase 1)
2. Call list_lineage_versions with since="2025-11-01Z" and until="2025-12-01Z"
3. Get version timestamps (e.g., ["2025-11-01T00:00:00Z", "2025-11-15T00:00:00Z", "2025-12-01T00:00:00Z"])
4. Call get_lineage_graph with dates=["2025-11-01T00:00:00Z", "2025-12-01T00:00:00Z"]
5. Call get_lineage_comparison with:
   • initial_lineage_assets: Same asset IDs used as lineage_ids in get_lineage_graph
   • compared_lineage_assets: All asset IDs returned by get_lineage_graph
   • base_version: "2025-12-01T00:00:00Z" (later date)
   • compared_version: "2025-11-01T00:00:00Z" (earlier date)

Important Guidelines

Lineage ID Validation

• CRITICAL: get_lineage_graph ONLY accepts 64-character hexadecimal lineage IDs
• Before calling get_lineage_graph, verify each lineage_id:
  • Length is exactly 64 characters
  • Contains only hexadecimal characters (0-9, a-f)
• If validation fails, use convert_asset_to_lineage_id or search_lineage_assets to get valid IDs

Hop Depth Best Practices

• Always inform the user about the 3-hop default limitation before executing
• Always confirm the user's preferred traversal depth
• Use hop values strategically:
  • "1": Immediate neighbors only
  • "3": Standard view (default)
  • "50": Deep traversal for complete pipeline view
  • "0": Exclude that direction entirely

Response Completeness

• Always return complete results - never truncate lineage assets or edges
• Always include the UI visualization URL in your response
• Always explain what the lineage graph shows in business terms, not just technical details

Error Handling

• If no assets found in Phase 1, suggest alternative search terms or filters
• If lineage graph is empty, explain possible reasons (asset not yet scanned, no relationships, etc.)
• If historical versions not available, explain that lineage versioning may not be enabled

Multi-Asset Queries

• When user provides multiple asset names, get lineage IDs for all of them
• Pass all lineage IDs together to get_lineage_graph to see relationships between them
• Use hop_up="50" and hop_down="50" for multi-asset queries to capture connections

Skill Completion

After completing the lineage exploration workflow:

1. Summarize what was discovered (sources, consumers, changes)
2. Provide the UI visualization link for further exploration
3. Offer to explore related aspects (e.g., "Would you like to see the data quality metrics for these assets?")
4. If historical comparison was performed, highlight the most significant changes

//: # (Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0\)) //: # (See the LICENSE file in the project root for license information.)