Data Quality Auditor
End-to-end data quality (DQ) practice: define DQ dimensions, write rule-based checks, detect schema drift, monitor freshness SLAs, respond to DQ incidents, build a maturity-graded program. Tool-agnostic — works whether you use Great Expectations, dbt tests, Soda Core, Monte Carlo, custom SQL, or hand-rolled scripts.
This skill is audit-focused, not pipeline-focused. For pipeline design, ETL, Spark/dbt, see engineering/senior-data-engineer.
Core Capabilities
- Six DQ dimensions — completeness, accuracy, consistency, timeliness/freshness, validity, uniqueness (plus integrity, conformity, reasonableness); at least one check per dimension at production stage.
- DQ check catalog — five categories (volume, freshness, schema, values, distribution) of ~50 specific check patterns applied per dataset.
- Schema drift detection — snapshot baseline schemas and diff added/removed/changed columns, types, and ordinals with severity.
- Freshness SLA monitoring — per-table max-age budgets with alerting-ready output.
- Incident response — severity classification (Sev1-4) and a 7-step playbook (acknowledge → quarantine → triage → contain → fix-forward → notify → post-incident) with recovery patterns.
- Maturity & governance — a five-level maturity model and anti-pattern catalog to grade and improve a DQ program.
When to Use
| Situation |
Skill applies |
| Setting up DQ from scratch on a new pipeline |
Yes — start with DQ dimensions + check catalog |
| Auditing existing pipelines for missing DQ |
Yes — dq_check_runner.py |
| Detecting schema drift in upstream sources |
Yes — schema_drift_detector.py |
| Monitoring freshness / SLA on data assets |
Yes — freshness_monitor.py |
| Responding to a DQ incident (bad data in prod) |
Yes — incident response playbook |
| Designing a DQ governance model |
Yes — DQ maturity model |
| Compliance evidence (SOC 2 PI1, GDPR, ISO 27001) |
Yes — checks produce auditable artifacts |
| Building data pipelines for the first time |
Use engineering/senior-data-engineer first |
Clarify First
Before running the audit, confirm these inputs. If any is unknown or vague, ASK — do not assume:
Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact.
The six DQ dimensions
Industry-standard taxonomy. Every dataset should have at least one check per dimension when at production stage.
| Dimension |
Question |
Example check |
| Completeness |
Are required fields populated? |
users.email IS NOT NULL — fail if > 0.1% nulls |
| Accuracy |
Do values match reality? |
Reconciliation against source-of-truth system; sample-based human review |
| Consistency |
Do values agree across systems / time? |
users.email in DB matches Salesforce; row count today within 5% of yesterday |
| Timeliness / Freshness |
Is data current to expectation? |
events_table.max(event_time) is < 1h old; pipeline runs SLA |
| Validity |
Do values conform to format / schema / business rules? |
Email regex matches; country code in ISO 3166-1; status in known enum |
| Uniqueness |
Are entities not duplicated? |
users.user_id is unique; no two rows with same (user_id, day) |
Some teams add: Integrity (referential — FKs resolve), Conformity (matches a published standard), Reasonableness (passes basic sanity checks beyond strict validity).
The DQ check catalog
Checks group into five categories applied per dataset — Volume, Freshness, Schema, Values, and Distribution. See the category summary and the full ~50-pattern catalog in references/dq-check-catalog.md.
Tools
| Tool |
Purpose |
Command |
dq_check_runner.py |
Run/profile DQ checks against tabular data; per-table pass/fail/warning with value vs threshold |
python scripts/dq_check_runner.py --data t.json --checks checks.json --format json |
schema_drift_detector.py |
Diff a current schema against a baseline snapshot (added/removed/changed columns, types, ordinals) |
python scripts/schema_drift_detector.py --baseline base.json --current cur.json |
freshness_monitor.py |
Check a freshness SLA: current age vs max-age budget, alerting-ready output |
python scripts/freshness_monitor.py --data t.json --column updated_at --max-age-min 60 |
All scripts: stdlib only, argparse CLI, JSON or human-readable output (see Scope re: live DB integration).
References
Load the reference that matches the task — keep this file lean and pull detail on demand:
- references/data-quality-dimensions.md — the 6 dimensions in depth: how to measure, threshold guidance, alerting strategy, and common pitfalls per dimension. Read when defining checks for a specific dimension.
- references/dq-check-catalog.md — the full catalog of ~50 specific check patterns with detection heuristics and tool snippets (Great Expectations / dbt / Soda / SQL). Read when writing concrete checks.
- references/dq-incident-response.md — the full incident playbook: severity classification, the 7-step response loop, recovery patterns (backfill, quarantine, DLQ, idempotent reprocessing, rollback), and post-incident writeup/notification templates. Read when responding to a DQ incident.
- references/dq-maturity-model.md — the five-level DQ maturity model (L0 reactive → L4 data-as-product) and what to invest in at each level. Read when grading or planning a DQ program.
- references/dq-workflows.md — the five end-to-end workflows (new dataset, schema drift, freshness SLA, auditing existing pipelines, post-incident improvement) with the exact script commands. Read when executing a concrete DQ task.
- references/dq-anti-patterns.md — the catalog of DQ anti-patterns to avoid. Read when reviewing an existing DQ program for smells.
Scope & Limitations
This skill covers:
- Auditing data quality across pipelines, warehouses, and stores against the six DQ dimensions.
- Rule-based check design, schema drift detection, and freshness SLA monitoring (tool-agnostic).
- DQ incident response and a maturity-graded governance program.
- Producing auditable DQ artifacts for compliance evidence (SOC 2 PI1, GDPR accuracy, ISO 27001).
This skill does NOT cover:
- Pipeline design, ETL, dbt/Spark implementation — use
engineering/senior-data-engineer.
- Live database querying; scripts are stdlib-only and read JSON inputs or simulate. For production, integrate your DB driver of choice (psycopg / mysqlclient / google-cloud-bigquery / etc.).
Related skills
engineering/senior-data-engineer — pipeline design, ETL, dbt, Sparkengineering/observability-designer — observability for data infrastructure (adjacent to DQ)engineering/chaos-engineering — DQ checks benefit from chaos testingra-qm-team/gdpr-dsgvo-expert — DQ underpins GDPR Art. 5(1)(d) "accuracy"ra-qm-team/soc2-compliance-expert — SOC 2 PI1 (Processing Integrity) requires DQ controls
1---2name: data-quality-auditor3description: Audit data quality across pipelines, warehouses, and stores. Use when designing a DQ program, defining DQ dimensions, building rule-based checks, detecting schema drift, monitoring freshness SLAs, or responding to a DQ incident.4license: MIT + Commons Clause5---6
7# Data Quality Auditor
8
9End-to-end data quality (DQ) practice: define DQ dimensions, write rule-based checks, detect schema drift, monitor freshness SLAs, respond to DQ incidents, build a maturity-graded program. Tool-agnostic — works whether you use Great Expectations, dbt tests, Soda Core, Monte Carlo, custom SQL, or hand-rolled scripts.
10
11This skill is audit-focused, not pipeline-focused. For pipeline design, ETL, Spark/dbt, see `engineering/senior-data-engineer`.
12
13## Core Capabilities
14
15- **Six DQ dimensions** — completeness, accuracy, consistency, timeliness/freshness, validity, uniqueness (plus integrity, conformity, reasonableness); at least one check per dimension at production stage.
16- **DQ check catalog** — five categories (volume, freshness, schema, values, distribution) of ~50 specific check patterns applied per dataset.
17- **Schema drift detection** — snapshot baseline schemas and diff added/removed/changed columns, types, and ordinals with severity.
18- **Freshness SLA monitoring** — per-table max-age budgets with alerting-ready output.
19- **Incident response** — severity classification (Sev1-4) and a 7-step playbook (acknowledge → quarantine → triage → contain → fix-forward → notify → post-incident) with recovery patterns.
20- **Maturity & governance** — a five-level maturity model and anti-pattern catalog to grade and improve a DQ program.
21
22## When to Use
23
24| Situation | Skill applies |
25|-----------|---------------|
26| Setting up DQ from scratch on a new pipeline | Yes — start with **DQ dimensions** + **check catalog** |
27| Auditing existing pipelines for missing DQ | Yes — `dq_check_runner.py` |
28| Detecting schema drift in upstream sources | Yes — `schema_drift_detector.py` |
29| Monitoring freshness / SLA on data assets | Yes — `freshness_monitor.py` |
30| Responding to a DQ incident (bad data in prod) | Yes — **incident response playbook** |
31| Designing a DQ governance model | Yes — **DQ maturity model** |
32| Compliance evidence (SOC 2 PI1, GDPR, ISO 27001) | Yes — checks produce auditable artifacts |
33| Building data pipelines for the first time | Use `engineering/senior-data-engineer` first |
34
35## Clarify First
36
37Before running the audit, confirm these inputs. If any is unknown or vague, ASK — do not assume:
38
39- [ ] **Dataset & target** — which table, pipeline, or store to audit (the data the scripts read via `--data`)
40- [ ] **Which check** — DQ rule checks, schema drift, or freshness SLA (selects `dq_check_runner.py` vs `schema_drift_detector.py` vs `freshness_monitor.py`)
41- [ ] **Thresholds & SLAs** — per-dimension pass thresholds and the freshness max-age budget (sets the pass/fail line and `--max-age-min`)
42
43Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact.
44
45## The six DQ dimensions
46
47Industry-standard taxonomy. Every dataset should have at least one check per dimension when at production stage.
48
49| Dimension | Question | Example check |
50|-----------|----------|---------------|
51| **Completeness** | Are required fields populated? | `users.email IS NOT NULL` — fail if > 0.1% nulls |
52| **Accuracy** | Do values match reality? | Reconciliation against source-of-truth system; sample-based human review |
53| **Consistency** | Do values agree across systems / time? | `users.email` in DB matches Salesforce; row count today within 5% of yesterday |
54| **Timeliness / Freshness** | Is data current to expectation? | `events_table.max(event_time)` is < 1h old; pipeline runs SLA |
55| **Validity** | Do values conform to format / schema / business rules? | Email regex matches; country code in ISO 3166-1; status in known enum |
56| **Uniqueness** | Are entities not duplicated? | `users.user_id` is unique; no two rows with same `(user_id, day)` |
57
58Some teams add: **Integrity** (referential — FKs resolve), **Conformity** (matches a published standard), **Reasonableness** (passes basic sanity checks beyond strict validity).
59
60## The DQ check catalog
61
62Checks group into five categories applied per dataset — **Volume**, **Freshness**, **Schema**, **Values**, and **Distribution**. See the category summary and the full ~50-pattern catalog in [references/dq-check-catalog.md](references/dq-check-catalog.md).
63
64## Tools
65
66| Tool | Purpose | Command |
67|------|---------|---------|
68| `dq_check_runner.py` | Run/profile DQ checks against tabular data; per-table pass/fail/warning with value vs threshold | `python scripts/dq_check_runner.py --data t.json --checks checks.json --format json` |
69| `schema_drift_detector.py` | Diff a current schema against a baseline snapshot (added/removed/changed columns, types, ordinals) | `python scripts/schema_drift_detector.py --baseline base.json --current cur.json` |
70| `freshness_monitor.py` | Check a freshness SLA: current age vs max-age budget, alerting-ready output | `python scripts/freshness_monitor.py --data t.json --column updated_at --max-age-min 60` |
71
72All scripts: stdlib only, argparse CLI, JSON or human-readable output (see Scope re: live DB integration).
73
74## References
75
76Load the reference that matches the task — keep this file lean and pull detail on demand:
77
78- **[references/data-quality-dimensions.md](references/data-quality-dimensions.md)** — the 6 dimensions in depth: how to measure, threshold guidance, alerting strategy, and common pitfalls per dimension. Read when defining checks for a specific dimension.
79- **[references/dq-check-catalog.md](references/dq-check-catalog.md)** — the full catalog of ~50 specific check patterns with detection heuristics and tool snippets (Great Expectations / dbt / Soda / SQL). Read when writing concrete checks.
80- **[references/dq-incident-response.md](references/dq-incident-response.md)** — the full incident playbook: severity classification, the 7-step response loop, recovery patterns (backfill, quarantine, DLQ, idempotent reprocessing, rollback), and post-incident writeup/notification templates. Read when responding to a DQ incident.
81- **[references/dq-maturity-model.md](references/dq-maturity-model.md)** — the five-level DQ maturity model (L0 reactive → L4 data-as-product) and what to invest in at each level. Read when grading or planning a DQ program.
82- **[references/dq-workflows.md](references/dq-workflows.md)** — the five end-to-end workflows (new dataset, schema drift, freshness SLA, auditing existing pipelines, post-incident improvement) with the exact script commands. Read when executing a concrete DQ task.
83- **[references/dq-anti-patterns.md](references/dq-anti-patterns.md)** — the catalog of DQ anti-patterns to avoid. Read when reviewing an existing DQ program for smells.
84
85## Scope & Limitations
86
87**This skill covers:**
88- Auditing data quality across pipelines, warehouses, and stores against the six DQ dimensions.
89- Rule-based check design, schema drift detection, and freshness SLA monitoring (tool-agnostic).
90- DQ incident response and a maturity-graded governance program.
91- Producing auditable DQ artifacts for compliance evidence (SOC 2 PI1, GDPR accuracy, ISO 27001).
92
93**This skill does NOT cover:**
94- Pipeline design, ETL, dbt/Spark implementation — use `engineering/senior-data-engineer`.
95- Live database querying; scripts are stdlib-only and read JSON inputs or simulate. For production, integrate your DB driver of choice (psycopg / mysqlclient / google-cloud-bigquery / etc.).
96
97## Related skills
98
99- `engineering/senior-data-engineer` — pipeline design, ETL, dbt, Spark
100- `engineering/observability-designer` — observability for data infrastructure (adjacent to DQ)
101- `engineering/chaos-engineering` — DQ checks benefit from chaos testing
102- `ra-qm-team/gdpr-dsgvo-expert` — DQ underpins GDPR Art. 5(1)(d) "accuracy"
103- `ra-qm-team/soc2-compliance-expert` — SOC 2 PI1 (Processing Integrity) requires DQ controls
borghei