Auto Data Discovery

Automated PII Discovery and Classification

Overview

Automated data discovery tools scan structured and unstructured data repositories to identify, classify, and catalogue personal data across the enterprise. Manual data inventories cannot keep pace with the volume, velocity, and variety of modern data processing. Automated discovery provides continuous visibility into where personal data resides, how it flows, and whether it is classified and protected according to policy. This skill covers implementation patterns for four leading platforms — Microsoft Purview, BigID, OneTrust DataDiscovery, and AWS Macie — with focus on scanning configuration, accuracy optimisation, and integration with privacy compliance workflows.

Platform Comparison

Implementation Pattern — Microsoft Purview

Architecture

Data Sources                    Microsoft Purview
┌─────────────┐                ┌───────────────────────────┐
│ Azure SQL   │──────scanner──►│ Data Map (metadata store)  │
│ SharePoint  │──────scanner──►│ Data Catalog (search/tag)  │
│ AWS S3      │──────scanner──►│ Data Estate Insights       │
│ On-prem SQL │──self-hosted──►│ Information Protection     │
│ Power BI    │──────scanner──►│ Data Loss Prevention (DLP) │
└─────────────┘                └───────────────────────────┘

Configuration Steps for Vanguard Financial Services

Step 1: Register Data Sources

• Register all Azure SQL databases, SharePoint sites, Azure Blob Storage accounts
• Install self-hosted integration runtime for on-premises SQL Server and file shares
• Register AWS S3 buckets via cross-cloud connector
• Register Power BI workspaces for report-level sensitivity scanning

Step 2: Configure Scanning Rules

• Enable built-in sensitive information types (SITs) for all EU jurisdictions:
  • UK National Insurance Number (pattern: [A-CEGHJ-PR-TW-Z]{2}\d{6}[A-D])
  • EU Passport Numbers (per Member State format)
  • International Banking Account Number (IBAN)
  • EU Tax Identification Numbers
  • EU Health Insurance Card Number
• Create custom SITs for Vanguard-specific identifiers:
  • Customer account number format: VFS-\d{10}
  • Employee ID format: EMP-[A-Z]{2}\d{6}
  • Investment portfolio reference: PF-\d{8}-[A-Z]{2}
• Configure confidence levels: HIGH for direct identifiers, MEDIUM for indirect, LOW for contextual matches
• Enable exact data match (EDM) for customer name verification against CRM database

Step 3: Set Scanning Schedule

• Full scan: Monthly (first Saturday of each month, 02:00 UTC)
• Incremental scan: Weekly (every Saturday, 02:00 UTC)
• On-demand scan: Triggered by new data source registration or incident response
• Estimated scan duration: 8 hours for full scan across 15 TB of data

Step 4: Configure Sensitivity Labels

• Map Purview sensitivity labels to Vanguard classification tiers:
  • Public → Purview label: Public
  • Internal → Purview label: General
  • Confidential → Purview label: Confidential
  • Restricted → Purview label: Highly Confidential (auto-applied to Art. 9/Art. 10 data)
• Enable auto-labelling policies: Apply labels based on detected SITs
• Configure label inheritance: Child items inherit parent container label unless higher

Step 5: DLP Policy Integration

• Create DLP policies triggered by sensitivity labels:
  • Restricted data: Block external sharing, require encryption, notify DPO
  • Confidential data: Warn on external sharing, audit log, require justification
  • Internal data: Audit log only
• Configure endpoint DLP for Windows devices (prevent copy to USB, print, screenshot for Restricted content)

Accuracy Tuning for Purview

Implementation Pattern — BigID

Architecture

BigID uses a distributed scanning architecture with correlation-based discovery:

Data Sources              BigID Platform
┌─────────────┐          ┌──────────────────────────┐
│ Databases   │──scan───►│ Discovery Engine          │
│ File Shares │──scan───►│ Correlation Engine (ML)   │
│ Cloud       │──scan───►│ Classification Engine     │
│ SaaS Apps   │──API────►│ Catalog & Inventory       │
│ Email       │──scan───►│ Privacy Rights Automation │
└─────────────┘          └──────────────────────────┘

Key Differentiator: Correlation-Based Identity Resolution

BigID's ML-based correlation engine identifies personal data by correlating data elements across sources to build identity profiles. This approach detects personal data that pattern matching alone would miss — for example, a customer ID in one system linked to a name in another.

Configuration for Vanguard

• Data source connections: Configure connectors for Salesforce CRM, Workday HR, ADP Payroll, SharePoint, Azure SQL, AWS S3, on-premises Oracle Data Warehouse
• Correlation rules: Enable identity correlation across CRM customer ID, HR employee ID, and payroll employee number
• Classification policies: Map BigID classifiers to GDPR categories (personal data, special category, criminal data, pseudonymised, anonymised)
• Scanning schedule: Incremental scans daily at 01:00 UTC; full correlation rebuild weekly

Implementation Pattern — OneTrust DataDiscovery

Architecture

OneTrust integrates discovery with its broader privacy management platform:

Data Sources              OneTrust Platform
┌─────────────┐          ┌──────────────────────────┐
│ Cloud/SaaS  │──API────►│ DataDiscovery Module      │
│ Databases   │──agent──►│ Data Mapping (Art. 30)    │
│ File Shares │──agent──►│ Assessment Automation     │
│ Endpoints   │──agent──►│ Consent Management        │
└─────────────┘          │ DSAR Automation           │
                         └──────────────────────────┘

Key Differentiator: Privacy Program Integration

OneTrust's value proposition is tight integration between discovery results and privacy program management — discovered personal data feeds directly into Art. 30 records, DPIA assessments, and DSAR fulfilment workflows.

Configuration for Vanguard

• Data source inventory: Use OneTrust's data mapping questionnaires to establish baseline, then deploy automated scanning to validate and enrich
• Integration with RoPA: Map discovered data categories to Art. 30 processing activity records
• DSAR integration: Configure discovery results to support automated data subject access request fulfilment (locate all data for a given identity across systems)
• Vendor discovery: Scan for personal data shared with third-party processors; reconcile with vendor management records

Implementation Pattern — AWS Macie

Architecture

Macie is purpose-built for S3 data discovery within AWS:

AWS Environment
┌─────────────────────────────────────┐
│  S3 Buckets    ──scan──►  Macie     │
│                          │          │
│  EventBridge  ◄──alerts──┘          │
│  Security Hub ◄──findings──┘        │
│  CloudWatch   ◄──metrics──┘         │
└─────────────────────────────────────┘

Configuration for Vanguard

• Bucket selection: Enable Macie for all S3 buckets in eu-west-1 (London) and eu-west-2 (Ireland) regions
• Custom data identifiers: Create identifiers for Vanguard-specific patterns (customer account numbers, employee IDs, portfolio references)
• Managed identifiers: Enable all EU-region managed data identifiers (UK NINO, IBAN, EU passport)
• Job scheduling: Scheduled classification jobs weekly; event-driven scan on new object upload for high-sensitivity buckets
• Findings integration: Route findings to AWS Security Hub for centralised monitoring; trigger Lambda function for automated label application

Scanning Schedule Best Practices

Accuracy Metrics and Tuning

Key Performance Indicators

Continuous Improvement Cycle

Month 1: Baseline scan → establish initial accuracy metrics
Month 2: Review false positives/negatives → tune rules and thresholds
Month 3: Re-scan → measure improvement
Month 4: Expand scope (new data sources) → re-baseline
Month 5: Review edge cases → create custom classifiers
Month 6: Accuracy audit by DPO → formal accuracy report
[Repeat cycle]

Integration Points

• data-inventory-mapping: Automated discovery feeds Art. 30 data inventory
• pii-in-unstructured: Specialized techniques for unstructured data augment platform capabilities
• data-labeling-system: Discovery results drive automated label application
• classification-policy: Discovery validates compliance with classification policy tiers
• data-lineage-tracking: Discovery combined with lineage provides complete data visibility