Data Quality Guardian
Implement comprehensive data quality frameworks with automated validation, anomaly detection, data contracts, and SLA monitoring.
Activation Triggers
Activate on: "data quality", "data validation", "Great Expectations", "data contract", "anomaly detection", "SLA", "freshness check", "schema validation", "data observability", "Soda", "elementary"
NOT for: dbt project layout → dbt-analytics-engineer | Schema evolution strategy → schema-evolution-manager | Data migration validation → data-migration-specialist
Quick Start
- Define data contracts — agree on schema, freshness, volume, and value ranges with upstream producers
- Implement tests — dbt tests for SQL models, Great Expectations for raw/external data
- Monitor freshness — alert when source tables are stale beyond SLA
- Detect anomalies — statistical checks for volume, distribution, and null rate changes
- Build quality dashboard — centralized view of all quality metrics across pipelines
Core Capabilities
| Domain |
Technologies |
| Testing |
dbt tests, Great Expectations 1.x, Soda Core 3.x |
| Observability |
elementary (dbt), Monte Carlo, Anomalo |
| Contracts |
Soda data contracts, dbt model contracts, Protobuf schemas |
| Anomaly Detection |
elementary anomaly monitors, custom z-score, Prophet |
| Alerting |
Slack/PagerDuty integration, SLA miss alerts |
Architecture Patterns
Multi-Layer Quality Checks
Raw Data (Landing)
↓
Layer 1: Schema Validation
- Column types match contract
- No unexpected NULLs in required fields
- Row count within expected range
↓
Layer 2: Business Rule Validation
- Referential integrity (FK relationships hold)
- accepted_values constraints
- Custom SQL assertions (e.g., revenue >= 0)
↓
Layer 3: Statistical Anomaly Detection
- Row count deviation from 7-day rolling average
- Null rate spike detection
- Distribution shift (KL divergence)
↓
Layer 4: Freshness & SLA Monitoring
- Source loaded within 2 hours
- Downstream models built within 4 hours
- Dashboard data <6 hours old
Data Contract YAML
# contracts/payments-contract.yml
contract:
name: stripe_payments
owner: payments-team
version: "2.0"
sla:
freshness: 2h # must be updated within 2 hours
volume_min: 1000 # at least 1000 rows per day
volume_max: 500000 # no more than 500k (anomaly if exceeded)
schema:
- name: payment_id
type: string
required: true
unique: true
- name: amount_cents
type: integer
required: true
checks:
- type: range
min: 0
max: 10000000 # $100k max
- name: status
type: string
required: true
checks:
- type: accepted_values
values: [succeeded, failed, pending, refunded]
- name: created_at
type: timestamp
required: true
checks:
- type: not_in_future
Anomaly Detection with elementary
# dbt model with elementary anomaly monitors
version: 2
models:
- name: fct_orders
tests:
- elementary.volume_anomalies:
timestamp_column: created_at
where: "created_at > dateadd(day, -30, current_date())"
sensitivity: 3 # z-score threshold
- elementary.column_anomalies:
column_name: total_amount
where: "created_at > dateadd(day, -30, current_date())"
- elementary.freshness_anomalies:
timestamp_column: _loaded_at
sensitivity: 2
Anti-Patterns
- Testing only in production — run quality checks in CI on sample data before merging; do not discover issues in production
- Alert fatigue — too many low-severity alerts get ignored; tier alerts: P1 (data loss), P2 (quality regression), P3 (informational)
- No contract with upstream — without an agreed contract, schema changes break pipelines silently
- Static thresholds only — hardcoded "row count > 1000" breaks during holidays; use statistical anomaly detection
- Quality checks at the end — validate at each layer (landing, staging, marts); catching errors early is cheaper
Quality Checklist
1---
2license: Apache-2.0
3name: data-quality-guardian
4description: "Great Expectations, dbt tests, anomaly detection, and data contracts for data quality. Activate on: data quality, data validation, Great Expectations, data contract, anomaly detection, SLA, freshness check, schema validation. NOT for: dbt model structure (use dbt-analytics-engineer), schema evolution (use schema-evolution-manager)."
5allowed-tools: Read,Write,Edit,Bash(npm:*,npx:*,python:*,dbt:*)
6category: Data & Analytics
7tags:
8 - data-quality
9 - testing
10 - great-expectations
11 - data-contracts
12 - anomaly-detection
13pairs-with:
14 - skill: dbt-analytics-engineer
15 reason: dbt tests are a primary data quality enforcement tool
16 - skill: data-lineage-tracker
17 reason: Quality issues traced via lineage to root cause
18 - skill: airflow-dag-orchestrator
19 reason: Quality checks run as orchestrated pipeline steps
20metadata:
21 source: https://skillmd.com/skills/curiositech/data-quality-guardian
22---
23
24# Data Quality Guardian
25
26Implement comprehensive data quality frameworks with automated validation, anomaly detection, data contracts, and SLA monitoring.
27
28## Activation Triggers
29
30**Activate on:** "data quality", "data validation", "Great Expectations", "data contract", "anomaly detection", "SLA", "freshness check", "schema validation", "data observability", "Soda", "elementary"
31
32**NOT for:** dbt project layout → `dbt-analytics-engineer` | Schema evolution strategy → `schema-evolution-manager` | Data migration validation → `data-migration-specialist`
33
34## Quick Start
35
361. **Define data contracts** — agree on schema, freshness, volume, and value ranges with upstream producers
372. **Implement tests** — dbt tests for SQL models, Great Expectations for raw/external data
383. **Monitor freshness** — alert when source tables are stale beyond SLA
394. **Detect anomalies** — statistical checks for volume, distribution, and null rate changes
405. **Build quality dashboard** — centralized view of all quality metrics across pipelines
41
42## Core Capabilities
43
44| Domain | Technologies |
45|--------|-------------|
46| **Testing** | dbt tests, Great Expectations 1.x, Soda Core 3.x |
47| **Observability** | elementary (dbt), Monte Carlo, Anomalo |
48| **Contracts** | Soda data contracts, dbt model contracts, Protobuf schemas |
49| **Anomaly Detection** | elementary anomaly monitors, custom z-score, Prophet |
50| **Alerting** | Slack/PagerDuty integration, SLA miss alerts |
51
52## Architecture Patterns
53
54### Multi-Layer Quality Checks
55
56```
57Raw Data (Landing)
58 ↓
59Layer 1: Schema Validation
60 - Column types match contract
61 - No unexpected NULLs in required fields
62 - Row count within expected range
63 ↓
64Layer 2: Business Rule Validation
65 - Referential integrity (FK relationships hold)
66 - accepted_values constraints
67 - Custom SQL assertions (e.g., revenue >= 0)
68 ↓
69Layer 3: Statistical Anomaly Detection
70 - Row count deviation from 7-day rolling average
71 - Null rate spike detection
72 - Distribution shift (KL divergence)
73 ↓
74Layer 4: Freshness & SLA Monitoring
75 - Source loaded within 2 hours
76 - Downstream models built within 4 hours
77 - Dashboard data <6 hours old
78```
79
80### Data Contract YAML
81
82```yaml
83# contracts/payments-contract.yml
84contract:
85 name: stripe_payments
86 owner: payments-team
87 version: "2.0"
88 sla:
89 freshness: 2h # must be updated within 2 hours
90 volume_min: 1000 # at least 1000 rows per day
91 volume_max: 500000 # no more than 500k (anomaly if exceeded)
92 schema:
93 - name: payment_id
94 type: string
95 required: true
96 unique: true
97 - name: amount_cents
98 type: integer
99 required: true
100 checks:
101 - type: range
102 min: 0
103 max: 10000000 # $100k max
104 - name: status
105 type: string
106 required: true
107 checks:
108 - type: accepted_values
109 values: [succeeded, failed, pending, refunded]
110 - name: created_at
111 type: timestamp
112 required: true
113 checks:
114 - type: not_in_future
115```
116
117### Anomaly Detection with elementary
118
119```yaml
120# dbt model with elementary anomaly monitors
121version: 2
122models:
123 - name: fct_orders
124 tests:
125 - elementary.volume_anomalies:
126 timestamp_column: created_at
127 where: "created_at > dateadd(day, -30, current_date())"
128 sensitivity: 3 # z-score threshold
129 - elementary.column_anomalies:
130 column_name: total_amount
131 where: "created_at > dateadd(day, -30, current_date())"
132 - elementary.freshness_anomalies:
133 timestamp_column: _loaded_at
134 sensitivity: 2
135```
136
137## Anti-Patterns
138
1391. **Testing only in production** — run quality checks in CI on sample data before merging; do not discover issues in production
1402. **Alert fatigue** — too many low-severity alerts get ignored; tier alerts: P1 (data loss), P2 (quality regression), P3 (informational)
1413. **No contract with upstream** — without an agreed contract, schema changes break pipelines silently
1424. **Static thresholds only** — hardcoded "row count > 1000" breaks during holidays; use statistical anomaly detection
1435. **Quality checks at the end** — validate at each layer (landing, staging, marts); catching errors early is cheaper
144
145## Quality Checklist
146
147- [ ] Data contracts defined for all critical sources (schema, freshness, volume)
148- [ ] dbt tests on every model: `unique`, `not_null`, `accepted_values`, `relationships`
149- [ ] Freshness monitoring with SLA thresholds and alerting
150- [ ] Anomaly detection on row counts, null rates, and value distributions
151- [ ] Quality dashboard shows current status across all pipelines
152- [ ] Alert severity tiered (P1/P2/P3) to prevent fatigue
153- [ ] Quality checks run in CI (pre-merge) and production (post-load)
154- [ ] Root cause analysis supported via data lineage integration
155- [ ] Monthly data quality review with trending metrics
156- [ ] Upstream teams notified within 15 minutes of contract violations