# Ds Data Privacy > "Applies privacy-preserving techniques including anonymization, differential privacy, encryption, and GDPR compliance for sensitive data" - Skill: `paulpas/ds-data-privacy` (Agent Skill) - Install (CLI): `npx skillmds@latest add paulpas/ds-data-privacy` - Raw SKILL.md: https://api.skillmd.com/api/skills/paulpas/ds-data-privacy/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: paulpas (https://skillmd.com/u/paulpas) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/paulpas/ds-data-privacy --- # Data Privacy Comprehensive guide to data privacy in machine learning and data science workflows. ## When to Use This Skill - Solving real-world data collection & ingestion problems - Building machine learning pipelines with data privacy - Implementing best practices for data privacy - Optimizing model performance using data privacy techniques - Learning industry-standard approaches to data privacy ## When NOT to Use This Skill - When using pre-built libraries without understanding underlying concepts - For toy problems that don't require data privacy rigor - When domain expertise in specific problem requires different approach - If your problem doesn't require the complexity this skill provides ## Purpose and Key Concepts Data Privacy is a critical component of the machine learning workflow. This skill covers: 1. **Theoretical foundations** — Mathematical principles and statistical concepts 2. **Practical implementation** — Working code examples and patterns 3. **Common pitfalls** — Mistakes to avoid and how to recover from them 4. **Best practices** — Industry-standard approaches and optimization techniques ## Core Workflow 1. **Understand the problem** — Clearly define what you're solving for 2. **Select approach** — Choose the right technique for your data and constraints 3. **Implement solution** — Write clean, tested code following best practices 4. **Validate results** — Verify your implementation with tests and validation 5. **Optimize performance** — Improve efficiency and accuracy incrementally ## Implementation Patterns ### Pattern 1: Basic Data Privacy ```python import pandas as pd import numpy as np from typing import Dict, Any TOKEN_SPACE_SIZE: int = 10000 def apply_basic_privacy(data: pd.DataFrame, epsilon: float = 1.0) -> Dict[str, Any]: """ Apply basic privacy-preserving transformations. Implements tokenization for categorical PII and Laplace noise for numerical data. """ if data.empty: raise ValueError("Input DataFrame cannot be empty") protected_data = data.copy() metrics: Dict[str, int] = {"columns_anonymized": 0, "noise_added": 0} for col in protected_data.columns: if protected_data[col].dtype == "object": protected_data[col] = protected_data[col].apply( lambda x: f"token_{abs(hash(str(x))) % TOKEN_SPACE_SIZE}" if pd.notna(x) else x ) metrics["columns_anonymized"] += 1 elif np.issubdtype(protected_data[col].dtype, np.number): sensitivity: float = protected_data[col].max() - protected_data[col].min() scale: float = sensitivity / epsilon noise: np.ndarray = np.random.laplace(0, scale, size=protected_data[col].shape) protected_data[col] = protected_data[col] + noise metrics["noise_added"] += 1 return {"protected_data": protected_data, "metrics": metrics} ``` ### Pattern 2: Production-Ready Data Privacy ```python import logging import pandas as pd import numpy as np from typing import Any, Dict, List logger = logging.getLogger(__name__) class DataPrivacyEngine: """Production-grade privacy engine implementing GDPR-compliant anonymization.""" def __init__(self, epsilon: float = 1.0, k_anonymity: int = 5) -> None: self.epsilon: float = epsilon self.k_anonymity: int = k_anonymity self._log: logging.Logger = logging.getLogger(self.__class__.__name__) def execute(self, data: pd.DataFrame, sensitive_cols: List[str]) -> Dict[str, Any]: """Execute privacy transformations on sensitive columns.""" if not isinstance(data, pd.DataFrame): raise TypeError("Expected pandas DataFrame") if data.empty: raise ValueError("Input data cannot be empty") protected: pd.DataFrame = data.copy() results: Dict[str, Any] = {"status": "success", "transformations_applied": []} for col in sensitive_cols: if col not in protected.columns: self._log.warning(f"Column {col} not found, skipping.") continue if protected[col].dtype == "object": protected[col] = protected[col].apply( lambda x: f"cat_{abs(hash(str(x))) % 100}" if pd.notna(x) else x ) results["transformations_applied"].append(f"tokenized_{col}") else: col_min: float = protected[col].min() col_max: float = protected[col].max() sensitivity: float = col_max - col_min scale: float = sensitivity / self.epsilon noise: np.ndarray = np.random.laplace(0, scale, size=len(protected)) protected[col] = protected[col] + noise results["transformations_applied"].append(f"dp_noised_{col}") results["protected_data"] = protected self._log.info(f"Privacy engine completed. Applied {len(results['transformations_applied'])} transformations.") return results ``` ## Best Practices - ✅ Always validate your implementation on test data - ✅ Document your assumptions and methodology - ✅ Use version control for reproducibility - ✅ Monitor performance metrics in production - ✅ Periodically review and update your approach - ✅ Test with edge cases and outliers - ✅ Log all significant operations for debugging ## Common Pitfalls | Pitfall | Problem | Solution | | --- --- ## Constraints ### MUST DO - Validate all data preprocessing steps are fit-only on training data, never on validation or test sets - Implement reproducible pipelines with fixed random seeds and deterministic operations where possible - Report model performance with confidence intervals via bootstrapping or cross-validation across multiple runs - Log all experiments with parameters, metrics, and artifacts using MLflow or equivalent tracking system ### MUST NOT DO - Do not evaluate a model on the same data used for training — always hold out a proper test set - Avoid overfitting to the validation set by limiting hyperparameter search iterations - Never use features that can only be computed at inference time (look-ahead bias) - Do not report single-run accuracy without statistical significance testing or error bars ## Live References > Authoritative documentation links for this skill's domain. The model follows markdown links at load time to resolve external references and inline content. - [Data Privacy — Wikipedia](https://en.wikipedia.org/wiki/Data_privacy) - [NIST Privacy Framework](https://www.nist.gov/privacy) - [GDPR Official Text (europa.eu)](https://gdpr.eu/) - [HIPAA Privacy Rule (HHS)](https://www.hhs.gov/hipaa/for-professionals/privacy/index.html) - [Data Protection Impact Assessment Guide (EDPB)](https://edpb.europa.eu/)