Implementing Pipeline Observability

Implementing Pipeline Observability

When to use

• Failures or stale data are discovered by stakeholders, not by your alerts.
• Setting up monitoring, SLAs, or on-call for data pipelines.
• Alerts are too noisy (fatigue) or too quiet (silent failures).
• Do NOT use for one-off incident debugging (use debugging-data-pipelines).

The four signals to monitor

1. Freshness — is data arriving within its SLA? (most incidents are lateness)
2. Volume — is row count within the expected range vs a trailing baseline?
3. Quality — do the data quality checks pass? (see implementing-data-quality-checks)
4. Run health — job success/failure, duration, and retry rate.

Workflow

- [ ] Define SLAs per critical dataset (freshness + volume)
- [ ] Emit run metrics: status, duration, rows in/out per run
- [ ] Add freshness + volume monitors with baseline-relative thresholds
- [ ] Route alerts to an owner with context and a runbook link
- [ ] Tune thresholds to cut false positives; track MTTD/MTTR

1. Set SLAs for datasets that feed decisions ("orders fresh within 3h, ±20% daily volume"). Without an SLA there is nothing to alert against.
2. Emit run metrics (status, duration, input/output row counts) to a store you can query and chart, not just scattered logs.
3. Monitor freshness and volume against trailing baselines, not fixed numbers, so seasonality doesn't trigger noise.
4. Make alerts actionable — every alert names the dataset, the breached SLA, the likely blast radius, and links a runbook. Route to a clear owner.
5. Tune — suppress duplicates, group related alerts, and set severities so people trust the pager.

Patterns

Freshness monitor:

-- Alert when the newest row is older than the SLA.
SELECT DATEDIFF('hour', MAX(ordered_at), CURRENT_TIMESTAMP()) AS hours_stale
FROM fct_orders
HAVING hours_stale > 3;

Volume anomaly vs trailing baseline — compare today's count to the mean of the prior N days and alert on a large relative deviation, catching partial loads and duplication that a fixed threshold misses.

Run metrics — record {pipeline, run_id, status, started_at, duration_s, rows_in, rows_out} per run; chart duration and row counts to spot drift before it becomes an incident. Tools: dbt artifacts, OpenLineage/Marquez, Elementary, Monte Carlo, or a simple metrics table.

Alert routing — critical → page the owning team with a runbook; warning → async channel. Deduplicate and group so one upstream failure isn't 50 pages.

Common pitfalls

• Alerting only on job failure — data can be stale or wrong while every job "succeeds"; monitor freshness/volume/quality too.
• Fixed thresholds on seasonal data — false positives train people to ignore alerts; use trailing baselines.
• Alerts with no owner or context — nobody acts; include dataset, SLA, impact, and a runbook.
• Over-alerting — fatigue causes real incidents to be missed; tune severity and dedupe.
• No metrics history — you can't see gradual regressions (creeping duration, shrinking volume) without stored run metrics.
• Monitoring everything equally — focus SLAs on datasets that drive decisions.