Data Validation

Data Validation Framework

Purpose

Comprehensive data validation framework for testing schema compliance, data quality, and referential integrity. Validates databases, APIs, data pipelines, and file formats. Generates data quality scorecards with anomaly detection.

Triggers

Use this skill when:

• "validate data quality"
• "check data integrity"
• "schema validation"
• "test data pipeline"
• "data quality report"
• "validate CSV"
• "check for data anomalies"
• "test ETL output"

When to Use

• Data pipeline deployment
• Database migration
• API response validation
• Report generation systems
• Data warehouse testing
• ML training data validation

When NOT to Use

• API endpoint testing (use api-contract-validator)
• Security testing (use security-test-suite)
• Performance testing (use performance-benchmark)

Core Instructions

Data Quality Dimensions

Validation Categories

Schema Definition

schema:
  tables:
    transactions:
      columns:
        - name: transaction_id
          type: string
          required: true
          unique: true
          pattern: "^TXN-[A-Z0-9]{10}$"

        - name: amount
          type: float
          required: true
          min: 0.01
          max: 1000000

        - name: status
          type: string
          required: true
          enum: [pending, completed, failed]

Templates

Data Quality Report

# Data Quality Report

**Source:** {source_type}
**Table:** {table_name}
**Generated:** {timestamp}

## Quality Scorecard

**Overall Score:** {score}/100 ({grade})

| Dimension | Score | Status |
| --------- | ----- | ------ |
| Completeness | {completeness} | {status_icon} |
| Accuracy | {accuracy} | {status_icon} |
| Consistency | {consistency} | {status_icon} |
| Uniqueness | {uniqueness} | {status_icon} |
| Freshness | {freshness} | {status_icon} |

## Data Summary

| Metric | Value |
| ------ | ----- |
| Total Rows | {total_rows} |
| Valid Rows | {valid_rows} ({valid_percent}%) |
| Invalid Rows | {invalid_rows} ({invalid_percent}%) |

## Issue Details

### {category} Issues

**{issue_id}:** {message}

- Column: `{column}`
- Affected rows: {row_count}
- Sample values: `{samples}`

Example

Input: Validate transactions CSV against schema

Output:

## Quality Scorecard

**Overall Score:** 87.3/100 (B)

| Dimension | Score | Status |
| --------- | ----- | ------ |
| Completeness | 95.0 | Pass |
| Accuracy | 88.5 | Pass |
| Consistency | 82.0 | Pass |
| Uniqueness | 100.0 | Pass |
| Freshness | 75.0 | Warn |

## Issue Details

### Accuracy Issues

**TYPE-amount:** Expected float, got string

- Column: `amount`
- Affected rows: 45
- Sample values: `"N/A", "pending", "TBD"`

Validation Checklist

• Schema definition matches expected structure
• All required columns validated
• Null thresholds appropriately set
• Foreign key references checked (if applicable)
• Anomaly detection parameters tuned
• Sample data reviewed for false positives
• Report includes actionable remediation

Related Skills

• api-contract-validator - For API response validation
• unit-test-generator - For data processing function tests
• test-health-monitor - For tracking validation trends