gitwalter Data Validation Skill
When to Use
Use this skill when you need to verify the integrity, type compliance, and business logic of data at any stage of the SDLC, particularly within ETL/ELT pipelines.
Prerequisites
- Defined schema (JSON Schema, SQL DDL, or Pydantic models).
- Access to the target data source or artifacts.
- Validated
data-patterns.jsonKnowledge Item.
Process
- Identify Schema: Select the target schema or definition to validate against.
- Execute Validation: Use the
validate_schematool to run the check. - Report Failures: Generate a summary of violations and their impact.
- Remediate: Fix the data or update the schema if it's outdated.
Best Practices
- Early Validation: Run validation as close to the source as possible.
- Automate: Integrate validation checks into CI/CD pipelines.
- Version Schemas: Always use versioned schemas to avoid regression.