# Data Quality Validation

> Data Quality Validation

- Skill: `kienbui1995/data-quality-validation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add kienbui1995/data-quality-validation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/kienbui1995/data-quality-validation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: kienbui1995 (https://skillmd.com/u/kienbui1995)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/kienbui1995/data-quality-validation

---


# Data Quality Validation

## When to Use
- Designing data quality checks for a pipeline
- Schema validation after ingestion
- Setting up monitoring for data freshness and completeness
- Preparing for GCP Professional Data Engineer exam

## Core Jobs

### 1. Data Quality Dimensions
- **Completeness** — no unexpected NULLs; all required fields populated
- **Accuracy** — values within expected ranges, valid formats
- **Consistency** — referential integrity, no duplicates, cross-table agreement
- **Freshness** — data arrived within expected SLA (lag monitoring)
- **Uniqueness** — no duplicate records on primary key

### 2. GCP Tools for Data Quality
| Tool | Use case |
|------|---------|
| **Dataform** | SQL-based transformation + data quality tests in BigQuery |
| **Cloud Data Quality (Cloud DQ)** | Rule-based DQ checks on BigQuery tables at scale |
| **Dataplex** | Data governance, discovery, quality across data lake |
| **Dataprep by Trifacta** | Visual data cleaning and profiling (no-code/low-code) |
| **BigQuery assertions** | Inline SQL checks in queries or scheduled queries |

### 3. Dataform Quality Tests
Use Dataform assertions to catch data quality issues:
- `assert_` prefix files run as data quality checks
- Returns rows that FAIL the assertion (empty result = passing)
- Integrate into CI/CD pipeline before promoting data to production

### 4. Schema Validation Patterns
- Validate schema on ingestion using Dataflow side outputs
- Reject malformed records to a dead-letter GCS bucket or Pub/Sub topic
- Use BigQuery table schema with REQUIRED mode for mandatory fields
- For JSON: validate against JSON Schema before writing

### 5. Data Freshness Monitoring
- Cloud Monitoring custom metrics for pipeline lag
- BigQuery scheduled queries to check MAX(created_at) vs current time
- Dataplex data quality tasks for ongoing freshness checks
- Alert via Cloud Monitoring → Pub/Sub → Cloud Functions for remediation

## Key Concepts
- **Data Catalog** — metadata management, tagging, business glossary, lineage
- **Dataplex** — unified data governance layer over GCS + BigQuery data lakes
- **Dataform** — version-controlled SQL transformations with built-in testing
- **INFORMATION_SCHEMA.TABLE_STORAGE** — monitor table sizes and freshness in BigQuery

## Checklist
- [ ] NULL checks on required fields?
- [ ] Range/format validation on critical columns?
- [ ] Duplicate detection on primary key?
- [ ] Dead-letter path for rejected records?
- [ ] Freshness monitoring with alerting?
- [ ] Schema registered in Data Catalog?

## Output Format
- 🔴 **Critical** — no validation on ingested data, no dead-letter for bad records
- 🟡 **Warning** — no freshness monitoring, no schema enforcement on ingestion
- 🟢 **Suggestion** — Dataform assertions for ongoing quality, Dataplex for governance

## Exam Tips
- **Dataform** = SQL-based ELT + testing in BigQuery (like dbt for GCP)
- **Dataplex** = governance + quality across the data lake (GCS + BigQuery)
- Dead-letter pattern = invalid records → separate GCS path for human review
- Data Catalog = metadata, not data quality (but integrates with Cloud DQ)
- Schema validation should happen at ingestion (Dataflow) not after writing to BigQuery