Data Pipeline Quality Check

Data Pipeline Quality Check Skill

Assess data quality for pipeline {{ pipeline_name }} ({{ source_system }} -> {{ target_system }}).

Workflow

Phase 1 — Pipeline Overview

PIPELINE INVENTORY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Pipeline: {{ pipeline_name }}
[ ] Source: {{ source_system }}
[ ] Target: {{ target_system }}
[ ] Schedule: [ ] Real-time  [ ] Micro-batch (___ min)  [ ] Batch (___ daily)
[ ] Daily data volume: ___ GB / ___ records
[ ] Pipeline technology: ___
[ ] Last successful run: ___
[ ] SLA: data available within ___ of source event

Phase 2 — Data Completeness

COMPLETENESS CHECKS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Record count reconciliation:
    - Source records (24h): ___
    - Target records (24h): ___
    - Delta: ___ (___ %)
    - Acceptable threshold: ___ %
[ ] Partition completeness:
    - All expected partitions present: [ ] YES  [ ] NO
    - Missing partitions: ___
[ ] Late-arriving data handling:
    - Strategy: [ ] Reprocess  [ ] Append  [ ] Ignore
    - Late data window: ___
[ ] Null analysis:
    - Required fields with nulls: ___
    - Null rate per field within threshold: [ ] YES  [ ] NO

Phase 3 — Data Accuracy

ACCURACY VALIDATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Schema validation:
    - Schema matches expected definition: [ ] YES  [ ] NO
    - Schema drift detected: [ ] YES  [ ] NO
    - New columns: ___
    - Removed columns: ___
    - Type changes: ___
[ ] Value range checks:
    - Numeric fields within expected bounds: [ ] YES
    - Date fields within valid ranges: [ ] YES
    - Enum fields contain valid values: [ ] YES
[ ] Referential integrity:
    - Foreign key relationships valid: [ ] YES  [ ] NO
    - Orphaned records: ___
[ ] Duplicate detection:
    - Duplicate records found: ___
    - Deduplication strategy: ___

Phase 4 — Data Freshness

FRESHNESS MONITORING
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] End-to-end latency:
    - Source event timestamp to target availability
    - P50: ___
    - P95: ___
    - P99: ___
    - SLA target: ___
    - SLA met: [ ] YES  [ ] NO
[ ] Staleness check:
    - Most recent record timestamp: ___
    - Expected freshness: ___
    - Freshness gap: ___
[ ] Processing time:
    - Average run duration: ___
    - Last run duration: ___

Phase 5 — Anomaly Detection

ANOMALY ANALYSIS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Volume anomalies:
    - Record count vs 7-day average: ___% deviation
    - Volume spike/drop detected: [ ] YES  [ ] NO
[ ] Distribution anomalies:
    - Key metric distributions within normal range: [ ] YES
    - Outliers identified: ___
[ ] Pattern anomalies:
    - Unexpected null patterns: [ ] YES  [ ] NO
    - Unexpected value distributions: [ ] YES  [ ] NO

Phase 6 — Lineage and Documentation

DATA LINEAGE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Source-to-target field mapping documented
[ ] Transformation logic documented
[ ] Data lineage tracked in catalog: [ ] YES  [ ] NO
[ ] Downstream consumers identified:
    - ___
    - ___
[ ] Data ownership:
    - Producer team: ___
    - Pipeline team: ___
    - Consumer team(s): ___
[ ] Data quality SLA documented: [ ] YES  [ ] NO

Counter-Rationalizations

Output Format

Produce a data pipeline quality report with:

1. Pipeline summary (source, target, schedule, volume)
2. Quality scorecard (completeness, accuracy, freshness scores)
3. Issues found (anomalies, schema drift, data loss)
4. SLA compliance (freshness and completeness vs targets)
5. Recommendations (monitoring improvements, quality gates, alerting)