Data Validation

Data Validation

Turn data expectations into executable checks with explicit failure behavior. Validation should protect a boundary, not merely produce a report.

Boundary

• Use data-quality to define ongoing trust indicators, service levels, and alert ownership.
• Use data-source-assessment when the source contract is still unknown.
• Use data-pipeline-testing to test transformation code and replay behavior.

Inputs

Establish:

1. Boundary being validated.
2. Declared grain and keys.
3. Required schema and semantic rules.
4. Allowed coercions and schema changes.
5. Batch, record, or table-level failure policy.
6. Evidence and retention requirements for rejected data.
7. Consumer tolerance for partial delivery.

Workflow

1. Write the contract independently of tools

Define:

• required, optional, and forbidden fields
• physical and semantic types
• nullability
• uniqueness and grain
• allowed categories and ranges
• cross-field rules
• referential requirements
• temporal ordering
• schema version and compatibility policy

Do not infer permanent contracts from one sample without domain confirmation.

2. Place checks at boundaries

Choose where each invariant is cheapest to diagnose and safest to enforce:

• record ingress
• batch ingestion
• post-normalization
• pre-load
• warehouse model
• publication

Validate raw shape before transformation and business meaning after normalization.

3. Choose the smallest implementation

Use framework-selection.md.

Prefer:

• language assertions or SQL for a few local invariants
• Pydantic for record and message boundaries
• Pandera for DataFrame-like schemas and checks
• Great Expectations when durable validation runs and shared artifacts justify it
• dbt data tests for warehouse model assertions

Do not add a framework when setup is larger than the contract.

4. Make failure semantics explicit

For every check, choose:

• block the entire batch
• quarantine invalid records
• publish with degraded status
• warn and continue
• retry after a transient dependency failure

State severity, owner, evidence retained, and replay behavior. Never silently drop invalid records.

5. Control coercion

Document accepted conversions such as string to date or integer widening. Reject ambiguous conversions such as locale-unknown dates or lossy numeric casts.

Report original value, attempted conversion, and error. Validation and normalization must not collapse into invisible mutation.

6. Handle schema evolution

Classify changes:

• additive compatible field
• required-field addition
• removal
• rename
• type widening
• type narrowing
• semantic or unit change
• grain or key change

Version contracts when consumers need transition time. A syntactically compatible change can still be semantically breaking.

7. Return actionable results

Each failed check should include:

• rule identifier and version
• boundary and dataset
• severity
• failed count and evaluated count
• representative failures with sensitive values redacted
• affected partition or cursor range
• owner and next action

Cap samples without hiding total failure counts.

8. Test the validators

Test valid, invalid, null, boundary, malformed, and schema-change cases. Verify quarantine, batch blocking, and replay behavior, not just rule predicates.

Output

# Data Validation Contract

## Boundary and Grain
...

## Rules
| ID | Boundary | Invariant | Severity | Failure behavior | Owner |
| --- | --- | --- | --- | --- | --- |

## Coercion Policy
- ...

## Schema Evolution Policy
- ...

## Validation Result Schema
- ...

## Validator Tests
- ...

Quality Gate

• Every rule protects a named boundary and consumer need.
• Grain, keys, and null semantics are explicit.
• Failure and quarantine behavior is testable.
• Coercions are deliberate and non-lossy.
• Schema evolution covers semantic changes.
• Error evidence is actionable and privacy-safe.