Data Lineage Guide
Overview
Use this skill any time the user wants to understand data flow, trace data origins, identify downstream impacts, or analyze how data pipelines have changed over time. This skill guides users through exploring data lineage relationships in a structured way, ensuring they understand the scope and depth of lineage traversal before executing queries.
Phase 0: Intent Detection and Entry Path Selection
Always start by understanding what the user wants to accomplish and which entry path is most appropriate:
Entry Path A: Direct Lineage Search (Lineage-First)
Use when the user mentions an asset name but doesn't specify a catalog/project context, or when they want to search across all lineage assets.
- Tool:
search_lineage_assets
- Best for: "Find the lineage for CUSTOMER_360", "What feeds into the sales table?"
- Supports: Optional filters for technology type and asset type
Entry Path B: Catalog Asset Lookup (Catalog-First)
Use when the user references a known catalog or project asset, or provides specific container context.
- Tools:
search_asset → convert_asset_to_lineage_id
- Best for: "Show lineage for the ORDERS table in the sales catalog", "Trace the customer_data asset in AgentTest project"
- Requires: Container context (catalog or project name)
Decision Logic:
- If user mentions "catalog" or "project" name → Use Path B (Catalog-First)
- If user only provides asset name without container → Use Path A (Lineage-First)
- If uncertain → Ask user to clarify or default to Path A
Phase 1: Asset Identification
Locate the starting asset(s) in the lineage graph using the appropriate entry path.
Path A: Direct Lineage Search
Path B: Catalog Asset Lookup
Important Validation:
- Lineage IDs must be exactly 64 hexadecimal characters
- If you receive a shorter ID or UUID, you MUST use
convert_asset_to_lineage_id to convert it
- Never proceed to Phase 2 without a valid 64-character lineage ID
Phase 2: Lineage Graph Traversal
Retrieve and present the upstream and downstream lineage relationships.
Critical: Inform User About Traversal Depth
BEFORE calling get_lineage_graph, you MUST:
Executing Lineage Graph Query
Presenting Lineage Results
Special Cases
Finding Path Between Two Assets:
- If user asks "trace from Asset A to Asset B" or "path between X and Y":
- Get lineage IDs for both assets (Phase 1)
- Call
get_lineage_graph with:
lineage_ids: [lineage_id_A, lineage_id_B]
hop_up: "50"
hop_down: "50"
ultimate: "" (empty string)
- Search the returned graph for the path connecting the two assets
Impact Analysis:
- If user asks "what would break if we changed X?":
- Focus on downstream traversal
- Set
hop_down to "50" for complete impact view
- Highlight all dependent tables, reports, and data products
Source Tracing:
- If user asks "where does this data come from originally?":
- Focus on upstream traversal
- Set
hop_up to "50" or use ultimate: "source"
- Identify the terminal source asset(s)
Phase 3: Historical Lineage Comparison (Optional)
Note: This phase depends on the list_lineage_versions tool which provides version timestamps for historical comparison.
Use this phase when the user wants to understand how the data pipeline has changed over time.
Natural Language Date Conversion:
- "since Friday" → Calculate the date of the most recent Friday and convert to ISO 8601
- "since last week" → Calculate 7 days ago from current date
- "since last month" → Calculate 30 days ago or first day of previous month
- "in the last 3 days" → Calculate 3 days ago from current date
- "this year" → Use current year start (e.g., "2026-01-01T00:00:00Z")
- Use the current time to calculate relative dates accurately
If user is comparing assets:
- Call
get_lineage_comparison with:
compared_lineage_assets: Asset IDs to compare
base_version: Later date (more recent version)
compared_version: Earlier date (older version)
If user is comparing graphs:
- Call
get_lineage_comparison with:
initial_lineage_assets: Asset IDs that were passed to get_lineage_graph as lineage_ids
compared_lineage_assets: All asset IDs returned by get_lineage_graph
base_version: Later date (more recent version)
compared_version: Earlier date (older version)
Historical Comparison Example Workflow
User: "Has anything changed in the pipeline for our revenue dashboard in the last month?"
- Find the revenue dashboard asset (Phase 1)
- Call
list_lineage_versions with since="2025-11-01Z" and until="2025-12-01Z"
- Get version timestamps (e.g., ["2025-11-01T00:00:00Z", "2025-11-15T00:00:00Z", "2025-12-01T00:00:00Z"])
- Call
get_lineage_graph with dates=["2025-11-01T00:00:00Z", "2025-12-01T00:00:00Z"]
- Call
get_lineage_comparison with:
initial_lineage_assets: Same asset IDs used as lineage_ids in get_lineage_graph
compared_lineage_assets: All asset IDs returned by get_lineage_graph
base_version: "2025-12-01T00:00:00Z" (later date)
compared_version: "2025-11-01T00:00:00Z" (earlier date)
Important Guidelines
Lineage ID Validation
- CRITICAL:
get_lineage_graph ONLY accepts 64-character hexadecimal lineage IDs
- Before calling
get_lineage_graph, verify each lineage_id:
- Length is exactly 64 characters
- Contains only hexadecimal characters (0-9, a-f)
- If validation fails, use
convert_asset_to_lineage_id or search_lineage_assets to get valid IDs
Hop Depth Best Practices
- Always inform the user about the 3-hop default limitation before executing
- Always confirm the user's preferred traversal depth
- Use hop values strategically:
- "1": Immediate neighbors only
- "3": Standard view (default)
- "50": Deep traversal for complete pipeline view
- "0": Exclude that direction entirely
Response Completeness
- Always return complete results - never truncate lineage assets or edges
- Always include the UI visualization URL in your response
- Always explain what the lineage graph shows in business terms, not just technical details
Error Handling
- If no assets found in Phase 1, suggest alternative search terms or filters
- If lineage graph is empty, explain possible reasons (asset not yet scanned, no relationships, etc.)
- If historical versions not available, explain that lineage versioning may not be enabled
Multi-Asset Queries
- When user provides multiple asset names, get lineage IDs for all of them
- Pass all lineage IDs together to
get_lineage_graph to see relationships between them
- Use hop_up="50" and hop_down="50" for multi-asset queries to capture connections
Skill Completion
After completing the lineage exploration workflow:
- Summarize what was discovered (sources, consumers, changes)
- Provide the UI visualization link for further exploration
- Offer to explore related aspects (e.g., "Would you like to see the data quality metrics for these assets?")
- If historical comparison was performed, highlight the most significant changes
//: # (Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0\))
//: # (See the LICENSE file in the project root for license information.)
1---2name : lineage3description : Use this skill to explore upstream/downstream data lineage and historical lineage changes through a guided 3-phase workflow - asset identification → lineage graph traversal → historical version comparison. Handles both direct lineage search and catalog-first lookup with ID conversion. Triggers when user asks about data relationships, sources, or consumers using phrases like - "what feeds", "what feeds into", "what feeds in", "where does X come from", "what sources", "what produces", "upstream", "downstream", "lineage", "impact analysis", "data pipeline", "trace this data", "what depends on", "what consumes", "what changed", "data flow", "source to target", "pipeline history", "lineage changes", "show lineage for", "get lineage of", "what is impacted"45metadata:6 source: https://skillmd.com/skills/ibm/lineage7---89# Data Lineage Guide1011## Overview1213Use this skill any time the user wants to understand data flow, trace data origins, identify downstream impacts, or analyze how data pipelines have changed over time. This skill guides users through exploring data lineage relationships in a structured way, ensuring they understand the scope and depth of lineage traversal before executing queries.1415## Phase 0: Intent Detection and Entry Path Selection1617Always start by understanding what the user wants to accomplish and which entry path is most appropriate:1819### Entry Path A: Direct Lineage Search (Lineage-First)20Use when the user mentions an asset name but doesn't specify a catalog/project context, or when they want to search across all lineage assets.21- **Tool**: `search_lineage_assets`22- **Best for**: "Find the lineage for CUSTOMER_360", "What feeds into the sales table?"23- **Supports**: Optional filters for technology type and asset type2425### Entry Path B: Catalog Asset Lookup (Catalog-First)26Use when the user references a known catalog or project asset, or provides specific container context.27- **Tools**: `search_asset` → `convert_asset_to_lineage_id`28- **Best for**: "Show lineage for the ORDERS table in the sales catalog", "Trace the customer_data asset in AgentTest project"29- **Requires**: Container context (catalog or project name)3031**Decision Logic**:32- If user mentions "catalog" or "project" name → Use Path B (Catalog-First)33- If user only provides asset name without container → Use Path A (Lineage-First)34- If uncertain → Ask user to clarify or default to Path A3536## Phase 1: Asset Identification3738Locate the starting asset(s) in the lineage graph using the appropriate entry path.3940### Path A: Direct Lineage Search4142<Steps>43<Step>441. Call `search_lineage_assets` with the asset name provided by the user. Include optional filters if the user specifies:45 - `technology_name`: Only if user mentions specific technology (e.g., "PostgreSQL", "Azure SQL")46 - `asset_type`: Only if user mentions specific type (e.g., "Table", "Column", "View")47 - `tag`: If user mentions tagged assets48 - `business_term` or `business_classification`: If user mentions governance metadata49 - `data_quality_operator` and `data_quality_value`: If user mentions quality thresholds50</Step>51<Step>522. The tool returns a list of matching lineage assets with their 64-character hexadecimal lineage IDs, names, types, and hierarchical paths.53</Step>54<Step>553. Present the matched assets to the user in a clear format showing:56 - Asset name57 - Asset type58 - Full hierarchical path (identity_key)59 - Parent asset information if available60</Step>61<Step>624. If multiple assets match, ask the user to confirm which asset(s) they want to explore. If only one asset matches, confirm it's the correct one before proceeding.63</Step>64<Step>655. Extract and save the lineage ID(s) from the confirmed asset(s) for Phase 2.66</Step>67</Steps>6869### Path B: Catalog Asset Lookup7071<Steps>72<Step>731. Call `search_asset` with:74 - `search_prompt`: The user's asset description75 - `container_type`: "catalog" or "project" based on user's context76</Step>77<Step>782. Present the search results to the user, showing asset names, IDs, and container information.79</Step>80<Step>813. Ask the user to confirm which asset they want to explore.82</Step>83<Step>844. Once confirmed, call `convert_asset_to_lineage_id` with:85 - `container_id`: The catalog or project ID from the search result86 - `asset_id`: The asset ID from the search result87</Step>88<Step>895. The tool returns the 64-character hexadecimal lineage ID required for lineage graph queries.90</Step>91<Step>926. Save the lineage ID for Phase 2.93</Step>94</Steps>9596**Important Validation**:97- Lineage IDs must be exactly 64 hexadecimal characters98- If you receive a shorter ID or UUID, you MUST use `convert_asset_to_lineage_id` to convert it99- Never proceed to Phase 2 without a valid 64-character lineage ID100101## Phase 2: Lineage Graph Traversal102103Retrieve and present the upstream and downstream lineage relationships.104105### Critical: Inform User About Traversal Depth106107**BEFORE calling `get_lineage_graph`, you MUST**:108109<Steps>110<Step>1111. Explain to the user that the default lineage traversal depth is **3 hops** in each direction (upstream and downstream). This means:112 - **3 hops upstream**: Shows 3 levels of data sources feeding into the asset113 - **3 hops downstream**: Shows 3 levels of consumers using the asset's data114 - **Limitation**: For complex or deeply nested pipelines, 3 hops may not capture the complete data flow115</Step>116<Step>1172. Present the user with traversal depth options:118 - **Immediate lineage (1-2 hops)**: Direct producers and consumers only119 - **Standard lineage (3 hops - default)**: Broader pipeline context, good for most use cases120 - **Full lineage (deep traversal - 50 hops)**: Complete path to ultimate sources and targets121 - Note: If the returned terminal asset ID matches the query asset ID, that asset IS the ultimate source/target122</Step>123<Step>1243. Ask the user which depth they prefer, or confirm if they want to proceed with the default 3 hops.125</Step>126</Steps>127128### Executing Lineage Graph Query129130<Steps>131<Step>1324. Based on the user's depth preference, call `get_lineage_graph` with:133 - `lineage_ids`: The 64-character lineage ID(s) from Phase 1 (can be a single ID or list of IDs)134 - `hop_up`: Number of upstream levels135 - "1" for immediate upstream136 - "3" for standard (default)137 - "50" for full upstream traversal138 - "0" if user only wants downstream139 - `hop_down`: Number of downstream levels140 - "1" for immediate downstream141 - "3" for standard (default)142 - "50" for full downstream traversal143 - "0" if user only wants upstream144 - `ultimate`: Optional parameter for ultimate source/target queries145 - "source" for ultimate source only146 - "target" for ultimate target only147 - "both" for both ultimate source and target148 - "" (empty string) when finding path between two assets149 - None for standard hop-based traversal150</Step>151<Step>1525. The tool returns:153 - `lineage_assets`: Complete list of assets in the lineage graph with metadata (name, type, tags, identity_key, parent info)154 - `edges_in_view`: Connections showing data flow (format: "edge from: AssetA, to: AssetB, relation: RelationType")155 - `url`: Direct link to visualize the lineage graph in the UI156</Step>157</Steps>158159### Presenting Lineage Results160161<Steps>162<Step>1636. Present the lineage graph results in a structured format:164 - **Upstream Sources**: List assets that feed data into the queried asset, organized by hop level if possible165 - **Downstream Consumers**: List assets that consume data from the queried asset, organized by hop level if possible166 - **Transformation Steps**: Highlight any transformation or processing assets in the pipeline167 - **Data Flow Connections**: Summarize key relationships from the edges_in_view168</Step>169<Step>1707. **ALWAYS include the UI visualization URL** in your response so users can explore the interactive lineage graph.171</Step>172<Step>1738. If the user requested ultimate source/target and the returned asset ID matches the query asset ID, explicitly state: "This asset IS the ultimate [source/target] - there are no further [upstream/downstream] dependencies."174</Step>175</Steps>176177### Special Cases178179**Finding Path Between Two Assets**:180- If user asks "trace from Asset A to Asset B" or "path between X and Y":181 1. Get lineage IDs for both assets (Phase 1)182 2. Call `get_lineage_graph` with:183 - `lineage_ids`: [lineage_id_A, lineage_id_B]184 - `hop_up`: "50"185 - `hop_down`: "50"186 - `ultimate`: "" (empty string)187 3. Search the returned graph for the path connecting the two assets188189**Impact Analysis**:190- If user asks "what would break if we changed X?":191 1. Focus on downstream traversal192 2. Set `hop_down` to "50" for complete impact view193 3. Highlight all dependent tables, reports, and data products194195**Source Tracing**:196- If user asks "where does this data come from originally?":197 1. Focus on upstream traversal198 2. Set `hop_up` to "50" or use `ultimate`: "source"199 3. Identify the terminal source asset(s)200201## Phase 3: Historical Lineage Comparison (Optional)202203**Note**: This phase depends on the `list_lineage_versions` tool which provides version timestamps for historical comparison.204205Use this phase when the user wants to understand how the data pipeline has changed over time.206207<Steps>208<Step>2091. Determine if the user wants historical comparison by looking for phrases like:210 - "what changed in the pipeline"211 - "has anything changed since [date]"212 - "compare lineage from [date] to [date]"213 - "pipeline changes over time"214 - "what's new in the data flow"215</Step>216<Step>2172. If historical comparison is requested, convert any natural language dates to ISO 8601 format, then call `list_lineage_versions` with:218 - `since`: Start date in ISO 8601 format (e.g., "2025-01-01T00:00:00Z" or "2025Z" for year)219 - `until`: End date in ISO 8601 format (e.g., "2025-12-31T23:59:59Z" or "2025Z" for year)220 221 **Natural Language Date Conversion**:222 - "since Friday" → Calculate the date of the most recent Friday and convert to ISO 8601223 - "since last week" → Calculate 7 days ago from current date224 - "since last month" → Calculate 30 days ago or first day of previous month225 - "in the last 3 days" → Calculate 3 days ago from current date226 - "this year" → Use current year start (e.g., "2026-01-01T00:00:00Z")227 - Use the current time to calculate relative dates accurately228</Step>229<Step>2303. The tool returns a list of available lineage version timestamps between the specified dates.231</Step>232<Step>2334. Select two versions to compare (typically first and last, or user-specified dates).234</Step>235<Step>2365. If user wants version comparison done for assets, call `search_lineage_assets` with `dates` parameter. Otherwise, if user wants comparison to be done for graph, use `get_lineage_graph` with the `dates` parameter:237 - Call with `dates` parameter containing the two version timestamps238 - This retrieves the lineage graph as it existed at those two points in time239</Step>240<Step>2416. Compare the two lineage graphs and identify changes:242 243 **If user is comparing assets:**244 - Call `get_lineage_comparison` with:245 - `compared_lineage_assets`: Asset IDs to compare246 - `base_version`: Later date (more recent version)247 - `compared_version`: Earlier date (older version)248 249 **If user is comparing graphs:**250 - Call `get_lineage_comparison` with:251 - `initial_lineage_assets`: Asset IDs that were passed to `get_lineage_graph` as `lineage_ids`252 - `compared_lineage_assets`: All asset IDs returned by `get_lineage_graph`253 - `base_version`: Later date (more recent version)254 - `compared_version`: Earlier date (older version)255</Step>256<Step>2577. Present a clear summary of changes:258 - List new assets with their types and roles259 - List removed assets and their previous roles260 - Highlight significant changes in data flow patterns261 - Note any changes in data quality or governance metadata262</Step>263</Steps>264265### Historical Comparison Example Workflow266267User: "Has anything changed in the pipeline for our revenue dashboard in the last month?"2682691. Find the revenue dashboard asset (Phase 1)2702. Call `list_lineage_versions` with since="2025-11-01Z" and until="2025-12-01Z"2713. Get version timestamps (e.g., ["2025-11-01T00:00:00Z", "2025-11-15T00:00:00Z", "2025-12-01T00:00:00Z"])2724. Call `get_lineage_graph` with dates=["2025-11-01T00:00:00Z", "2025-12-01T00:00:00Z"]2735. Call `get_lineage_comparison` with:274 - `initial_lineage_assets`: Same asset IDs used as `lineage_ids` in `get_lineage_graph`275 - `compared_lineage_assets`: All asset IDs returned by `get_lineage_graph`276 - `base_version`: "2025-12-01T00:00:00Z" (later date)277 - `compared_version`: "2025-11-01T00:00:00Z" (earlier date)278279## Important Guidelines280281### Lineage ID Validation282- **CRITICAL**: `get_lineage_graph` ONLY accepts 64-character hexadecimal lineage IDs283- Before calling `get_lineage_graph`, verify each lineage_id:284 - Length is exactly 64 characters285 - Contains only hexadecimal characters (0-9, a-f)286- If validation fails, use `convert_asset_to_lineage_id` or `search_lineage_assets` to get valid IDs287288### Hop Depth Best Practices289- **Always inform the user** about the 3-hop default limitation before executing290- **Always confirm** the user's preferred traversal depth291- Use hop values strategically:292 - "1": Immediate neighbors only293 - "3": Standard view (default)294 - "50": Deep traversal for complete pipeline view295 - "0": Exclude that direction entirely296297### Response Completeness298- **Always return complete results** - never truncate lineage assets or edges299- **Always include the UI visualization URL** in your response300- **Always explain** what the lineage graph shows in business terms, not just technical details301302### Error Handling303- If no assets found in Phase 1, suggest alternative search terms or filters304- If lineage graph is empty, explain possible reasons (asset not yet scanned, no relationships, etc.)305- If historical versions not available, explain that lineage versioning may not be enabled306307### Multi-Asset Queries308- When user provides multiple asset names, get lineage IDs for all of them309- Pass all lineage IDs together to `get_lineage_graph` to see relationships between them310- Use hop_up="50" and hop_down="50" for multi-asset queries to capture connections311312## Skill Completion313314After completing the lineage exploration workflow:3151. Summarize what was discovered (sources, consumers, changes)3162. Provide the UI visualization link for further exploration3173. Offer to explore related aspects (e.g., "Would you like to see the data quality metrics for these assets?")3184. If historical comparison was performed, highlight the most significant changes319320---321322[//]: # (Copyright [2026] [IBM])323[//]: # (Licensed under the Apache License, Version 2.0 \(http://www.apache.org/licenses/LICENSE-2.0\))324[//]: # (See the LICENSE file in the project root for license information.)