Amey-Thakur Data retention
Retention is a promise in two directions: data the business needs stays available, and data past its purpose actually disappears, everywhere, provably. Most organizations keep the first promise and quietly break the second.
Method
- Classify data with owners and clocks. Per dataset: class (PII, financial, telemetry, derived), business owner, retention period with its reason (regulation cites, business need), and the clock start (creation, last activity, contract end). This table, kept where lineage metadata lives (see data-lineage), is the spec everything else implements.
- Tier before you delete. Hot store for the active window, warm/cold object tiers for the tail (see cloud-storage-selection), delete at the end: lifecycle rules automate the transitions. Aggregate-then-drop is the analytics pattern: keep the rollups (see warehouse-modeling) with long retention, expire the raw events; most "we might need it" arguments are satisfied by aggregates at 1% of the cost.
- Build deletion as a pipeline, not a script. Scheduled jobs that find expired data by the classification table, delete across all stores (primary DB, warehouse, lake partitions, search indexes, caches, logs), and emit an audit record of what was deleted when and by which rule. Partition-aligned layouts (see data-partitioning) turn expiry into cheap partition drops instead of row-by-row deletes.
- Handle erasure requests (GDPR/CCPA) as their own path. Subject-keyed deletion within the statutory window: locate via lineage/catalog everywhere the subject's data lives, delete or anonymize, and handle the backup problem: either restore-time re-deletion (documented, tested) or crypto-shredding (per-subject keys destroyed on request; see event-sourcing for the immutable-log variant). Log the fulfillment as evidence.
- Implement legal hold as an override switch. Holds pause deletion for named scopes (custodian, matter, dataset) above all retention rules; release resumes normal expiry. The deletion pipeline checks holds before every run; a hold discovered after deletion is a sanctions conversation, so this check fails closed.
- Audit the machinery quarterly. Sample expired data and prove it is gone from every store; reconcile the classification table against reality (new tables appear constantly; see data-quality-checks for the scanning muscle); review retention periods with legal yearly. Deletion code that never gets audited is deletion theater.
Boundaries
- Retention policy content (how long, what law) is a legal decision; this skill builds the enforcement, not the numbers. Get the table signed off.
- Derived data (models, embeddings, aggregates) containing personal data inherits erasure obligations; whether an aggregate is sufficiently anonymized is a privacy-review question (see pii-handling), not an engineering assumption.
- Soft-delete flags are UX, not retention compliance; flagged rows still exist for every purpose a regulator cares about.