# Ds Data Quality > "Implements data validation, cleaning, outlier detection, and quality assurance techniques to ensure reliable datasets for model training" - Skill: `paulpas/ds-data-quality` (Agent Skill) - Install (CLI): `npx skillmds@latest add paulpas/ds-data-quality` - Raw SKILL.md: https://api.skillmd.com/api/skills/paulpas/ds-data-quality/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - License: MIT - Author: paulpas (https://skillmd.com/u/paulpas) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/paulpas/ds-data-quality --- # Data Quality Comprehensive guide to data quality in machine learning and data science workflows. ## When to Use This Skill - Solving real-world data collection & ingestion problems - Building machine learning pipelines with data quality - Implementing best practices for data quality - Optimizing model performance using data quality techniques - Learning industry-standard approaches to data quality ## When NOT to Use This Skill - When using pre-built libraries without understanding underlying concepts - For toy problems that don't require data quality rigor - When domain expertise in specific problem requires different approach - If your problem doesn't require the complexity this skill provides ## Purpose and Key Concepts Data Quality is a critical component of the machine learning workflow. This skill covers: 1. **Theoretical foundations** — Mathematical principles and statistical concepts 2. **Practical implementation** — Working code examples and patterns 3. **Common pitfalls** — Mistakes to avoid and how to recover from them 4. **Best practices** — Industry-standard approaches and optimization techniques ## Core Workflow 1. **Understand the problem** — Clearly define what you're solving for 2. **Select approach** — Choose the right technique for your data and constraints 3. **Implement solution** — Write clean, tested code following best practices 4. **Validate results** — Verify your implementation with tests and validation 5. **Optimize performance** — Improve efficiency and accuracy incrementally ## Implementation Patterns ### Pattern 1: Basic Data Quality ```python import pandas as pd import numpy as np from typing import Dict, Any def basic_data_quality_check(df: pd.DataFrame) -> Dict[str, Any]: """ Perform basic data quality checks on a DataFrame. Checks for missing values, duplicates, data types, and basic statistics. """ if not isinstance(df, pd.DataFrame): raise TypeError("Input must be a pandas DataFrame") quality_report = { "total_rows": len(df) "total_columns": len(df.columns) "missing_values": df.isnull().sum().to_dict() "duplicate_rows": int(df.duplicated().sum()) "data_types": df.dtypes.astype(str).to_dict() "numeric_summary": {} } for col in df.select_dtypes(include=[np.number]).columns: quality_report["numeric_summary"][col] = { "mean": float(df[col].mean()) "std": float(df[col].std()) "min": float(df[col].min()) "max": float(df[col].max()) "null_count": int(df[col].isnull().sum()) } return quality_report ``` ### Pattern 2: Production-Ready Data Quality ```python import logging import pandas as pd import numpy as np from typing import Any, Dict, List, Optional from sklearn.impute import SimpleImputer from sklearn.preprocessing import StandardScaler logger = logging.getLogger(__name__) class ProductionDataQuality: """ Production-grade data quality handler following DRY principles. Handles validation, imputation, scaling, and outlier detection. """ def __init__(self, missing_threshold: float = 0.5, outlier_std: float = 3.0): self.missing_threshold = missing_threshold self.outlier_std = outlier_std self.imputer = SimpleImputer(strategy="median") self.scaler = StandardScaler() self.quality_issues: List[str] = [] def execute(self, data: pd.DataFrame) -> Dict[str, Any]: """Execute comprehensive data quality pipeline.""" if data.empty: raise ValueError("Input DataFrame cannot be empty") issues = [] clean_data = data.copy() missing_cols = clean_data.columns[clean_data.isnull().mean() > self.missing_threshold] if len(missing_cols) > 0: issues.append(f"Dropping columns with >{self.missing_threshold*100}% missing: {list(missing_cols)}") clean_data.drop(columns=missing_cols, inplace=True) numeric_cols = clean_data.select_dtypes(include=[np.number]).columns if len(numeric_cols) > 0: clean_data[numeric_cols] = self.imputer.fit_transform(clean_data[numeric_cols]) for col in numeric_cols: z_scores = np.abs((clean_data[col] - clean_data[col].mean()) / clean_data[col].std()) outlier_mask = z_scores > self.outlier_std if outlier_mask.any(): issues.append(f"Detected {outlier_mask.sum()} outliers in column '{col}'") clean_data.loc[outlier_mask, col] = clean_data[col].median() if len(numeric_cols) > 0: clean_data[numeric_cols] = self.scaler.fit_transform(clean_data[numeric_cols]) self.quality_issues = issues return { "cleaned_data": clean_data "issues_found": issues "rows_processed": len(data) "columns_retained": len(clean_data.columns) } ``` ### BAD vs GOOD Example ```python # BAD: Bypasses error handling, uses magic numbers, and mutates input silently def bad_quality_check(df): df.dropna() df = df[(df['val'] > -999) & (df['val'] < 999)] return df # GOOD: Explicit validation, configurable thresholds, and proper return structure def good_quality_check(df: pd.DataFrame, threshold: float = 0.5) -> pd.DataFrame: if not isinstance(df, pd.DataFrame): raise TypeError("Input must be a DataFrame") if df.empty: raise ValueError("DataFrame cannot be empty") clean_df = df.dropna() numeric_cols = clean_df.select_dtypes(include=[np.number]).columns for col in numeric_cols: q1, q3 = clean_df[col].quantile(0.25), clean_df[col].quantile(0.75) iqr = q3 - q1 clean_df = clean_df[(clean_df[col] >= q1 - 1.5 * iqr) & (clean_df[col] <= q3 + 1.5 * iqr)] return clean_df ``` ## Best Practices - ✅ Always validate your implementation on test data - ✅ Document your assumptions and methodology - ✅ Use version control for reproducibility - ✅ Monitor performance metrics in production - ✅ Periodically review and update your approach - ✅ Test with edge cases and outliers - ✅ Log all significant operations for debugging ## Common Pitfalls | Pitfall | Problem | Solution | | --- --- ## Constraints ### MUST DO - Validate all data preprocessing steps are fit-only on training data, never on validation or test sets - Implement reproducible pipelines with fixed random seeds and deterministic operations where possible - Report model performance with confidence intervals via bootstrapping or cross-validation across multiple runs - Log all experiments with parameters, metrics, and artifacts using MLflow or equivalent tracking system ### MUST NOT DO - Do not evaluate a model on the same data used for training — always hold out a proper test set - Avoid overfitting to the validation set by limiting hyperparameter search iterations - Never use features that can only be computed at inference time (look-ahead bias) - Do not report single-run accuracy without statistical significance testing or error bars ## Live References > Authoritative documentation links for this skill's domain. The model follows markdown links at load time to resolve external references and inline content. - [Data Quality — Wikipedia](https://en.wikipedia.org/wiki/Data_quality) - [Great Expectations Documentation](https://docs.greatexpectations.io/) - [NIST Data Quality Guide](https://www.nist.gov/itl/div898/excel/data-quality) - [Data Quality Framework (TDWI)](https://tdwi.org/research/2019/03/27/data-quality-framework.aspx) - [Kaggle Data Quality Tutorial](https://www.kaggle.com/learn/data-cleaning)