AI Model Privacy Audit
Overview
AI model privacy auditing is the systematic assessment of whether trained ML models leak information about their training data. Models can memorize individual training records, enabling adversaries to extract personal data, determine dataset membership, reconstruct input features, or infer sensitive attributes. This skill implements a comprehensive model privacy audit methodology using established attack techniques and tools (ML Privacy Meter, ART, Foolbox) to quantify privacy leakage before deployment and periodically during operation. The audit results feed directly into the AI DPIA risk assessment and inform mitigation measure selection.
Privacy Attack Taxonomy
1. Training Data Extraction
Objective: Extract verbatim or near-verbatim records from the model's training data.
| Attack Vector |
Description |
Target Models |
| Prompt-based extraction |
Craft prompts that cause LLMs to regurgitate training data |
Language models, generative models |
| Canary extraction |
Insert known canary strings into training data and test if model reproduces them |
Any model (testing methodology) |
| Gradient-based extraction |
Use model gradients to reconstruct training inputs |
Models with accessible gradients |
| Generative reconstruction |
Use the model as an oracle to iteratively reconstruct training samples |
GANs, VAEs, diffusion models |
Risk Factors Increasing Extraction Likelihood:
- Large model capacity relative to training data size (overfitting)
- Training data containing duplicated or near-duplicated records
- Longer training duration (more epochs)
- Lower regularisation
- Models with high output granularity (logits, probabilities)
Testing Methodology:
- Insert canary records with unique identifiers into training data
- Train the model
- Attempt extraction through various prompting strategies
- Measure extraction success rate (percentage of canaries recovered)
- Threshold: extraction rate should be below 0.1% for acceptable risk
2. Membership Inference
Objective: Determine whether a specific record was in the model's training set.
| Attack Type |
Method |
Computational Cost |
| Shadow model attack |
Train shadow models on similar data, build a binary classifier on model outputs |
High — requires training multiple shadow models |
| Metric-based attack |
Use model confidence, loss, or entropy to distinguish members from non-members |
Low — single model query per sample |
| Label-only attack |
Use predicted labels (no confidence scores) to infer membership |
Medium — requires multiple queries |
| Likelihood ratio attack (LiRA) |
Compare per-sample loss to reference distributions |
High — most accurate, requires multiple models |
ML Privacy Meter Implementation:
- Population metric-based attack: compares target model's loss on a sample against population loss distribution
- Reference metric-based attack: uses reference models to compute per-sample metrics
- Shadow model attack: trains shadow models and uses the attack model to classify members
Testing Methodology:
- Partition data: training set (members) and held-out set (non-members)
- Run membership inference attacks using ML Privacy Meter
- Measure attack success: true positive rate at low false positive rate (TPR@FPR=0.1%, 1%)
- Generate ROC curves per sample and aggregate
- Threshold: TPR@1%FPR should be below 5% for acceptable privacy
3. Model Inversion
Objective: Reconstruct input features from model outputs.
| Attack Type |
Method |
Target |
| Confidence-based inversion |
Iteratively optimise input to maximise model confidence for a known label |
Classification models |
| Gradient-based inversion |
Use model gradients to reconstruct inputs from outputs |
White-box models |
| GAN-based inversion |
Train a GAN to invert model outputs to input space |
Face recognition, image classifiers |
Testing Methodology:
- Select target classes or individuals
- Run inversion attacks with various initializations
- Measure reconstruction quality (SSIM, PSNR for images; cosine similarity for embeddings)
- Assess re-identification risk: can reconstructed data identify specific individuals?
- Threshold: reconstruction similarity should be below 0.3 (SSIM) for acceptable risk
4. Attribute Inference
Objective: Infer sensitive attributes not present in the model's output.
| Attack Type |
Description |
| Correlation exploitation |
Use correlated features to infer sensitive attributes from model behaviour |
| Partial knowledge attack |
Attacker knows some attributes and uses model to infer remaining sensitive ones |
| Group inference |
Determine statistical properties of training subgroups |
Testing Methodology:
- Identify sensitive attributes (Art. 9 categories) that may be correlated with model features
- Train attack models to predict sensitive attributes from model outputs
- Measure inference accuracy for each sensitive attribute
- Compare against random baseline
- Threshold: inference accuracy should not exceed random baseline + 10%
Audit Methodology
Phase 1: Audit Scoping (Days 1-3)
- Define audit scope: which models, what deployment context, what threat model
- Identify assets: training data, model artefacts, deployment infrastructure
- Define threat model: who are the adversaries, what access do they have?
- Black-box: API access only (queries and responses)
- Grey-box: API access plus model architecture knowledge
- White-box: Full access to model weights and architecture
- Select attacks based on threat model and model type
- Define success criteria (acceptable leakage thresholds)
- Obtain audit authorisation from model owner and legal
Phase 2: Environment Setup (Days 4-7)
- Set up isolated audit environment (no production data leakage)
- Install audit tools: ML Privacy Meter, ART (Adversarial Robustness Toolbox), custom scripts
- Obtain model access (API endpoint or model weights depending on threat model)
- Prepare member/non-member datasets for membership inference
- Prepare canary data for extraction testing
- Configure monitoring to log all audit queries
Phase 3: Attack Execution (Days 8-18)
For each selected attack:
- Configure attack parameters
- Execute attack against the target model
- Collect results (success rates, confidence intervals)
- Vary attack parameters to find worst-case leakage
- Document: attack configuration, results, computational cost
Phase 4: Analysis and Reporting (Days 19-25)
- Aggregate results across all attacks
- Calculate privacy risk scores per attack type
- Identify high-risk data subsets (records most vulnerable to extraction)
- Cross-reference with DPIA risk register
- Generate audit report with:
- Executive summary
- Attack results per category
- Risk assessment with GDPR alignment
- Recommended mitigations
- Residual risk after proposed mitigations
Phase 5: Remediation Validation (Days 26-30)
- If mitigations are applied (differential privacy, output perturbation, etc.)
- Re-run key attacks to validate mitigation effectiveness
- Document residual leakage post-mitigation
- Compare against acceptable thresholds
- Issue final audit certificate or remediation requirements
Privacy Leakage Thresholds
| Metric |
Acceptable |
Elevated |
Unacceptable |
| Membership inference TPR@1%FPR |
< 5% |
5-15% |
> 15% |
| Training data extraction rate |
< 0.1% |
0.1-1% |
> 1% |
| Model inversion SSIM |
< 0.3 |
0.3-0.6 |
> 0.6 |
| Attribute inference accuracy above baseline |
< 10% |
10-25% |
> 25% |
Mitigation Measures
| Mitigation |
Attacks Mitigated |
Trade-off |
| Differential privacy (DP-SGD) |
All — provides mathematical guarantee |
Model accuracy reduction (calibrate epsilon) |
| Training data deduplication |
Extraction, membership inference |
One-time preprocessing cost |
| Regularisation (dropout, weight decay) |
Membership inference, overfitting-related leakage |
May affect model performance |
| Output perturbation |
Model inversion, attribute inference |
Reduces output precision |
| Confidence score rounding |
Metric-based membership inference |
Minor output precision loss |
| Model distillation |
Extraction, membership inference |
Requires additional training |
| Rate limiting |
All query-based attacks |
Affects legitimate use |
| Input/output PII filtering |
Extraction of PII from generative models |
May affect model utility |
Tools and Frameworks
| Tool |
Purpose |
Source |
| ML Privacy Meter |
Membership inference auditing |
github.com/privacytrustlab/ml_privacy_meter |
| IBM ART |
Adversarial robustness and privacy testing |
github.com/Trusted-AI/adversarial-robustness-toolbox |
| TensorFlow Privacy |
Differential privacy training |
github.com/tensorflow/privacy |
| Opacus |
PyTorch differential privacy |
github.com/pytorch/opacus |
| Google DP Library |
Differential privacy algorithms |
github.com/google/differential-privacy |
| Foolbox |
Adversarial attack library |
github.com/bethgelab/foolbox |
Enforcement Relevance
Model privacy auditing is not explicitly required by the GDPR or AI Act, but is effectively mandated through:
- Art. 35 DPIA: Risk assessment for AI systems must evaluate privacy leakage risks — auditing is the standard methodology
- Art. 32 Security: Appropriate technical measures to ensure security of processing — privacy auditing validates these measures
- AI Act Art. 9: Risk management for high-risk AI requires identification and mitigation of privacy risks
- AI Act Art. 15: Accuracy, robustness, and cybersecurity requirements — privacy attacks are a cybersecurity concern
- EDPB Guidelines 04/2025: Controllers must assess whether AI models have effectively anonymised training data — auditing tests this claim
Integration Points
- ai-dpia: Audit results feed into DPIA Phase 3 risk assessment
- ai-data-retention: Audit validates whether deletion from training data is effective
- ai-deployment-checklist: Pre-deployment privacy audit is a checklist requirement
- ai-federated-learning: Federated learning models require distributed privacy auditing
1---2name: ai-model-privacy-audit3description: Conducts privacy auditing of AI models including training data extraction testing, membership inference attacks, model inversion testing, and attribute inference assessment. Uses ML Privacy Meter and related tools to quantify privacy leakage. Keywords: model audit, membership inference, privacy meter, model inversion, training data extraction.4license: Apache-2.05---6# AI Model Privacy Audit
7
8## Overview
9
10AI model privacy auditing is the systematic assessment of whether trained ML models leak information about their training data. Models can memorize individual training records, enabling adversaries to extract personal data, determine dataset membership, reconstruct input features, or infer sensitive attributes. This skill implements a comprehensive model privacy audit methodology using established attack techniques and tools (ML Privacy Meter, ART, Foolbox) to quantify privacy leakage before deployment and periodically during operation. The audit results feed directly into the AI DPIA risk assessment and inform mitigation measure selection.
11
12## Privacy Attack Taxonomy
13
14### 1. Training Data Extraction
15
16**Objective**: Extract verbatim or near-verbatim records from the model's training data.
17
18| Attack Vector | Description | Target Models |
19|---------------|-------------|---------------|
20| Prompt-based extraction | Craft prompts that cause LLMs to regurgitate training data | Language models, generative models |
21| Canary extraction | Insert known canary strings into training data and test if model reproduces them | Any model (testing methodology) |
22| Gradient-based extraction | Use model gradients to reconstruct training inputs | Models with accessible gradients |
23| Generative reconstruction | Use the model as an oracle to iteratively reconstruct training samples | GANs, VAEs, diffusion models |
24
25**Risk Factors Increasing Extraction Likelihood**:
26- Large model capacity relative to training data size (overfitting)
27- Training data containing duplicated or near-duplicated records
28- Longer training duration (more epochs)
29- Lower regularisation
30- Models with high output granularity (logits, probabilities)
31
32**Testing Methodology**:
331. Insert canary records with unique identifiers into training data
342. Train the model
353. Attempt extraction through various prompting strategies
364. Measure extraction success rate (percentage of canaries recovered)
375. Threshold: extraction rate should be below 0.1% for acceptable risk
38
39### 2. Membership Inference
40
41**Objective**: Determine whether a specific record was in the model's training set.
42
43| Attack Type | Method | Computational Cost |
44|-------------|--------|-------------------|
45| Shadow model attack | Train shadow models on similar data, build a binary classifier on model outputs | High — requires training multiple shadow models |
46| Metric-based attack | Use model confidence, loss, or entropy to distinguish members from non-members | Low — single model query per sample |
47| Label-only attack | Use predicted labels (no confidence scores) to infer membership | Medium — requires multiple queries |
48| Likelihood ratio attack (LiRA) | Compare per-sample loss to reference distributions | High — most accurate, requires multiple models |
49
50**ML Privacy Meter Implementation**:
51- Population metric-based attack: compares target model's loss on a sample against population loss distribution
52- Reference metric-based attack: uses reference models to compute per-sample metrics
53- Shadow model attack: trains shadow models and uses the attack model to classify members
54
55**Testing Methodology**:
561. Partition data: training set (members) and held-out set (non-members)
572. Run membership inference attacks using ML Privacy Meter
583. Measure attack success: true positive rate at low false positive rate (TPR@FPR=0.1%, 1%)
594. Generate ROC curves per sample and aggregate
605. Threshold: TPR@1%FPR should be below 5% for acceptable privacy
61
62### 3. Model Inversion
63
64**Objective**: Reconstruct input features from model outputs.
65
66| Attack Type | Method | Target |
67|-------------|--------|--------|
68| Confidence-based inversion | Iteratively optimise input to maximise model confidence for a known label | Classification models |
69| Gradient-based inversion | Use model gradients to reconstruct inputs from outputs | White-box models |
70| GAN-based inversion | Train a GAN to invert model outputs to input space | Face recognition, image classifiers |
71
72**Testing Methodology**:
731. Select target classes or individuals
742. Run inversion attacks with various initializations
753. Measure reconstruction quality (SSIM, PSNR for images; cosine similarity for embeddings)
764. Assess re-identification risk: can reconstructed data identify specific individuals?
775. Threshold: reconstruction similarity should be below 0.3 (SSIM) for acceptable risk
78
79### 4. Attribute Inference
80
81**Objective**: Infer sensitive attributes not present in the model's output.
82
83| Attack Type | Description |
84|-------------|-------------|
85| Correlation exploitation | Use correlated features to infer sensitive attributes from model behaviour |
86| Partial knowledge attack | Attacker knows some attributes and uses model to infer remaining sensitive ones |
87| Group inference | Determine statistical properties of training subgroups |
88
89**Testing Methodology**:
901. Identify sensitive attributes (Art. 9 categories) that may be correlated with model features
912. Train attack models to predict sensitive attributes from model outputs
923. Measure inference accuracy for each sensitive attribute
934. Compare against random baseline
945. Threshold: inference accuracy should not exceed random baseline + 10%
95
96## Audit Methodology
97
98### Phase 1: Audit Scoping (Days 1-3)
99
1001. Define audit scope: which models, what deployment context, what threat model
1012. Identify assets: training data, model artefacts, deployment infrastructure
1023. Define threat model: who are the adversaries, what access do they have?
103 - Black-box: API access only (queries and responses)
104 - Grey-box: API access plus model architecture knowledge
105 - White-box: Full access to model weights and architecture
1064. Select attacks based on threat model and model type
1075. Define success criteria (acceptable leakage thresholds)
1086. Obtain audit authorisation from model owner and legal
109
110### Phase 2: Environment Setup (Days 4-7)
111
1121. Set up isolated audit environment (no production data leakage)
1132. Install audit tools: ML Privacy Meter, ART (Adversarial Robustness Toolbox), custom scripts
1143. Obtain model access (API endpoint or model weights depending on threat model)
1154. Prepare member/non-member datasets for membership inference
1165. Prepare canary data for extraction testing
1176. Configure monitoring to log all audit queries
118
119### Phase 3: Attack Execution (Days 8-18)
120
121For each selected attack:
1221. Configure attack parameters
1232. Execute attack against the target model
1243. Collect results (success rates, confidence intervals)
1254. Vary attack parameters to find worst-case leakage
1265. Document: attack configuration, results, computational cost
127
128### Phase 4: Analysis and Reporting (Days 19-25)
129
1301. Aggregate results across all attacks
1312. Calculate privacy risk scores per attack type
1323. Identify high-risk data subsets (records most vulnerable to extraction)
1334. Cross-reference with DPIA risk register
1345. Generate audit report with:
135 - Executive summary
136 - Attack results per category
137 - Risk assessment with GDPR alignment
138 - Recommended mitigations
139 - Residual risk after proposed mitigations
140
141### Phase 5: Remediation Validation (Days 26-30)
142
1431. If mitigations are applied (differential privacy, output perturbation, etc.)
1442. Re-run key attacks to validate mitigation effectiveness
1453. Document residual leakage post-mitigation
1464. Compare against acceptable thresholds
1475. Issue final audit certificate or remediation requirements
148
149## Privacy Leakage Thresholds
150
151| Metric | Acceptable | Elevated | Unacceptable |
152|--------|-----------|----------|-------------|
153| Membership inference TPR@1%FPR | < 5% | 5-15% | > 15% |
154| Training data extraction rate | < 0.1% | 0.1-1% | > 1% |
155| Model inversion SSIM | < 0.3 | 0.3-0.6 | > 0.6 |
156| Attribute inference accuracy above baseline | < 10% | 10-25% | > 25% |
157
158## Mitigation Measures
159
160| Mitigation | Attacks Mitigated | Trade-off |
161|-----------|-------------------|-----------|
162| Differential privacy (DP-SGD) | All — provides mathematical guarantee | Model accuracy reduction (calibrate epsilon) |
163| Training data deduplication | Extraction, membership inference | One-time preprocessing cost |
164| Regularisation (dropout, weight decay) | Membership inference, overfitting-related leakage | May affect model performance |
165| Output perturbation | Model inversion, attribute inference | Reduces output precision |
166| Confidence score rounding | Metric-based membership inference | Minor output precision loss |
167| Model distillation | Extraction, membership inference | Requires additional training |
168| Rate limiting | All query-based attacks | Affects legitimate use |
169| Input/output PII filtering | Extraction of PII from generative models | May affect model utility |
170
171## Tools and Frameworks
172
173| Tool | Purpose | Source |
174|------|---------|--------|
175| ML Privacy Meter | Membership inference auditing | github.com/privacytrustlab/ml_privacy_meter |
176| IBM ART | Adversarial robustness and privacy testing | github.com/Trusted-AI/adversarial-robustness-toolbox |
177| TensorFlow Privacy | Differential privacy training | github.com/tensorflow/privacy |
178| Opacus | PyTorch differential privacy | github.com/pytorch/opacus |
179| Google DP Library | Differential privacy algorithms | github.com/google/differential-privacy |
180| Foolbox | Adversarial attack library | github.com/bethgelab/foolbox |
181
182## Enforcement Relevance
183
184Model privacy auditing is not explicitly required by the GDPR or AI Act, but is effectively mandated through:
185
186- **Art. 35 DPIA**: Risk assessment for AI systems must evaluate privacy leakage risks — auditing is the standard methodology
187- **Art. 32 Security**: Appropriate technical measures to ensure security of processing — privacy auditing validates these measures
188- **AI Act Art. 9**: Risk management for high-risk AI requires identification and mitigation of privacy risks
189- **AI Act Art. 15**: Accuracy, robustness, and cybersecurity requirements — privacy attacks are a cybersecurity concern
190- **EDPB Guidelines 04/2025**: Controllers must assess whether AI models have effectively anonymised training data — auditing tests this claim
191
192## Integration Points
193
194- **ai-dpia**: Audit results feed into DPIA Phase 3 risk assessment
195- **ai-data-retention**: Audit validates whether deletion from training data is effective
196- **ai-deployment-checklist**: Pre-deployment privacy audit is a checklist requirement
197- **ai-federated-learning**: Federated learning models require distributed privacy auditing
aibot88