Data Lineage Tracking
Purpose
Map and document the complete lifecycle of personal data through organizational systems, from initial collection point through all transformations, transfers, and storage locations to final deletion or anonymization.
Prerequisites
- Data inventory or asset register identifying systems processing personal data
- Network architecture documentation showing system interconnections
- Access to system metadata, ETL configurations, and API documentation
- Records of Processing Activities (RoPA) under GDPR Article 30
Workflow
Step 1: Define Lineage Scope
Determine scope boundaries for the lineage mapping exercise:
- Data categories: Identify which personal data categories to trace (identity data, contact data, financial data, behavioral data, special category data under Art. 9)
- System boundaries: Define which systems are in-scope (production databases, data warehouses, analytics platforms, third-party SaaS, backup systems)
- Temporal scope: Determine whether to map current-state lineage only or include historical data flows
- Legal basis mapping: Link each processing activity in the lineage to its GDPR Article 6 lawful basis
Step 2: Identify Data Sources (Collection Points)
Document every point where personal data enters the organization:
- Direct collection: Web forms, mobile apps, point-of-sale terminals, customer service interactions, paper forms digitized via scanning
- Indirect collection: Third-party data providers, publicly available sources, data brokers, partner organizations
- Derived data: Data generated through processing (risk scores, customer segments, behavioral profiles)
- Inferred data: Data inferred from other data points (creditworthiness, health predictions, preferences)
For each source, record:
- Source identifier and type
- Data categories collected (referencing Art. 30(1)(c) categories)
- Legal basis under Art. 6(1) and, if applicable, Art. 9(2)
- Information provided to data subjects per Art. 13 or Art. 14
- Volume and frequency of collection
Step 3: Map Data Transformations
Document every transformation applied to personal data:
- ETL processes: Extract-Transform-Load pipelines moving data between systems
- Aggregation: Grouping individual records into summary statistics
- Pseudonymization: Replacing identifiers with tokens per Art. 4(5) and Recital 26
- Anonymization: Irreversible de-identification per WP29 Opinion 05/2014
- Enrichment: Combining data from multiple sources to create enriched profiles
- Format conversion: Changing data formats (CSV to JSON, database migration)
For each transformation, record:
- Input data categories and source system
- Transformation logic description
- Output data categories and destination system
- Whether transformation changes the identifiability of data subjects
- Retention period at destination per Art. 5(1)(e) storage limitation
Step 4: Document Data Flows and Transfers
Map all movements of personal data between systems and parties:
- Internal flows: Between departments, systems, databases within the organization
- Processor transfers: To data processors under Art. 28 agreements
- Third-country transfers: Cross-border transfers requiring Art. 44-49 safeguards
- Third-party disclosures: To independent controllers (regulators, partners, law enforcement)
For each flow, record:
- Source and destination system/entity
- Transfer mechanism (API, file transfer, database replication, manual export)
- Legal safeguard for international transfers (SCCs, BCRs, adequacy decision)
- Encryption in transit and at rest
- Frequency and volume
Step 5: Map Data Storage and Retention
Document where personal data resides at each stage:
- Primary storage: Production databases, CRM systems, HRIS
- Secondary storage: Data warehouses, analytics databases, reporting systems
- Archival storage: Long-term archives, cold storage, compliance archives
- Backup storage: Disaster recovery systems, backup tapes, cloud backup
- Temporary storage: Caches, message queues, log files, session storage
For each storage location, record:
- Storage technology and location (on-premises, cloud region)
- Retention period and legal basis for retention
- Access controls and encryption
- Deletion or anonymization mechanism at end of retention period
Step 6: Implement Automated Lineage Discovery
Deploy tooling to automate lineage tracking:
- Database-level lineage: Query log analysis, column-level lineage from SQL parsing
- Application-level lineage: API call tracing, service mesh observability
- Pipeline-level lineage: ETL tool metadata (Apache Airflow lineage backend, dbt documentation)
- Infrastructure-level lineage: Network flow logs, data lake audit trails
Use the scripts/process.py helper to parse system metadata and generate lineage graphs.
Step 7: Integrate with RoPA and Compliance
Link lineage data to GDPR compliance documentation:
- Art. 30 RoPA: Each lineage path should map to a processing activity in the RoPA
- DPIA triggers: Flag lineage paths involving Art. 35(3) processing (large-scale profiling, systematic monitoring, special categories)
- Data subject rights: Use lineage to locate all data for DSAR responses (Art. 15 access, Art. 17 erasure, Art. 20 portability)
- Breach impact scoping: Use lineage to determine affected data subjects and categories during incident response per Art. 33(3)
Step 8: Maintain and Validate
Establish ongoing lineage maintenance:
- Change management: Update lineage when new systems, data flows, or processing activities are introduced
- Periodic validation: Quarterly review to verify lineage accuracy against actual system behavior
- Stakeholder review: Annual sign-off from data owners, system architects, and DPO
- Completeness check: Cross-reference lineage against data inventory and RoPA to identify gaps
Verification
1---2name: data-lineage-tracking3description: >-4 Implements data lineage tracking for privacy compliance including origin tracking,5 transformation logging, access auditing, deletion verification, and cross-system6 lineage graphs. Covers source-to-sink mapping, GDPR Art. 30 RoPA integration,7 automated lineage discovery, and breach impact scoping. Keywords: data lineage,8 data provenance, data flow mapping, transformation logging, deletion verification.9license: Apache-2.010metadata:11 source: https://skillmd.com/skills/mukul975-2/data-lineage-tracking12 author: mukul97513 version: "1.0"14 domain: privacy15 subdomain: data-classification16 tags: "data-lineage, data-provenance, data-flow-mapping, transformation-logging, deletion-verification"17---1819# Data Lineage Tracking2021## Purpose2223Map and document the complete lifecycle of personal data through organizational systems, from initial collection point through all transformations, transfers, and storage locations to final deletion or anonymization.2425## Prerequisites2627- Data inventory or asset register identifying systems processing personal data28- Network architecture documentation showing system interconnections29- Access to system metadata, ETL configurations, and API documentation30- Records of Processing Activities (RoPA) under GDPR Article 303132## Workflow3334### Step 1: Define Lineage Scope3536Determine scope boundaries for the lineage mapping exercise:37381. **Data categories**: Identify which personal data categories to trace (identity data, contact data, financial data, behavioral data, special category data under Art. 9)392. **System boundaries**: Define which systems are in-scope (production databases, data warehouses, analytics platforms, third-party SaaS, backup systems)403. **Temporal scope**: Determine whether to map current-state lineage only or include historical data flows414. **Legal basis mapping**: Link each processing activity in the lineage to its GDPR Article 6 lawful basis4243### Step 2: Identify Data Sources (Collection Points)4445Document every point where personal data enters the organization:46471. **Direct collection**: Web forms, mobile apps, point-of-sale terminals, customer service interactions, paper forms digitized via scanning482. **Indirect collection**: Third-party data providers, publicly available sources, data brokers, partner organizations493. **Derived data**: Data generated through processing (risk scores, customer segments, behavioral profiles)504. **Inferred data**: Data inferred from other data points (creditworthiness, health predictions, preferences)5152For each source, record:53- Source identifier and type54- Data categories collected (referencing Art. 30(1)(c) categories)55- Legal basis under Art. 6(1) and, if applicable, Art. 9(2)56- Information provided to data subjects per Art. 13 or Art. 1457- Volume and frequency of collection5859### Step 3: Map Data Transformations6061Document every transformation applied to personal data:62631. **ETL processes**: Extract-Transform-Load pipelines moving data between systems642. **Aggregation**: Grouping individual records into summary statistics653. **Pseudonymization**: Replacing identifiers with tokens per Art. 4(5) and Recital 26664. **Anonymization**: Irreversible de-identification per WP29 Opinion 05/2014675. **Enrichment**: Combining data from multiple sources to create enriched profiles686. **Format conversion**: Changing data formats (CSV to JSON, database migration)6970For each transformation, record:71- Input data categories and source system72- Transformation logic description73- Output data categories and destination system74- Whether transformation changes the identifiability of data subjects75- Retention period at destination per Art. 5(1)(e) storage limitation7677### Step 4: Document Data Flows and Transfers7879Map all movements of personal data between systems and parties:80811. **Internal flows**: Between departments, systems, databases within the organization822. **Processor transfers**: To data processors under Art. 28 agreements833. **Third-country transfers**: Cross-border transfers requiring Art. 44-49 safeguards844. **Third-party disclosures**: To independent controllers (regulators, partners, law enforcement)8586For each flow, record:87- Source and destination system/entity88- Transfer mechanism (API, file transfer, database replication, manual export)89- Legal safeguard for international transfers (SCCs, BCRs, adequacy decision)90- Encryption in transit and at rest91- Frequency and volume9293### Step 5: Map Data Storage and Retention9495Document where personal data resides at each stage:96971. **Primary storage**: Production databases, CRM systems, HRIS982. **Secondary storage**: Data warehouses, analytics databases, reporting systems993. **Archival storage**: Long-term archives, cold storage, compliance archives1004. **Backup storage**: Disaster recovery systems, backup tapes, cloud backup1015. **Temporary storage**: Caches, message queues, log files, session storage102103For each storage location, record:104- Storage technology and location (on-premises, cloud region)105- Retention period and legal basis for retention106- Access controls and encryption107- Deletion or anonymization mechanism at end of retention period108109### Step 6: Implement Automated Lineage Discovery110111Deploy tooling to automate lineage tracking:1121131. **Database-level lineage**: Query log analysis, column-level lineage from SQL parsing1142. **Application-level lineage**: API call tracing, service mesh observability1153. **Pipeline-level lineage**: ETL tool metadata (Apache Airflow lineage backend, dbt documentation)1164. **Infrastructure-level lineage**: Network flow logs, data lake audit trails117118Use the `scripts/process.py` helper to parse system metadata and generate lineage graphs.119120### Step 7: Integrate with RoPA and Compliance121122Link lineage data to GDPR compliance documentation:1231241. **Art. 30 RoPA**: Each lineage path should map to a processing activity in the RoPA1252. **DPIA triggers**: Flag lineage paths involving Art. 35(3) processing (large-scale profiling, systematic monitoring, special categories)1263. **Data subject rights**: Use lineage to locate all data for DSAR responses (Art. 15 access, Art. 17 erasure, Art. 20 portability)1274. **Breach impact scoping**: Use lineage to determine affected data subjects and categories during incident response per Art. 33(3)128129### Step 8: Maintain and Validate130131Establish ongoing lineage maintenance:1321331. **Change management**: Update lineage when new systems, data flows, or processing activities are introduced1342. **Periodic validation**: Quarterly review to verify lineage accuracy against actual system behavior1353. **Stakeholder review**: Annual sign-off from data owners, system architects, and DPO1364. **Completeness check**: Cross-reference lineage against data inventory and RoPA to identify gaps137138## Verification139140- [ ] All personal data categories have documented source-to-sink lineage paths141- [ ] Each lineage node includes legal basis, retention period, and access controls142- [ ] Transformation steps document identifiability impact143- [ ] Cross-border transfers identified with Art. 44-49 safeguards documented144- [ ] Lineage integrates with Art. 30 RoPA entries145- [ ] Automated discovery covers at least database and pipeline layers146- [ ] Quarterly validation schedule established with responsible owners assigned