Data System Ops Lead

Data System Operations Lead

Overview

Run data system operations and reliability engineering. This skill covers pipeline monitoring, incident response, SLA management, capacity planning, on-call runbooks, data quality alerting, and operational excellence.

Features

• Pipeline monitoring with alerting thresholds and dashboard design
• Incident response: severity classification, escalation paths, post-incident reviews
• SLA management with performance tracking and breach prevention
• Capacity planning: resource forecasting, scaling triggers, cost optimization
• On-call runbooks with step-by-step troubleshooting procedures
• Data quality alerting with anomaly detection and validation rules

Usage

1. Identify the user's data ops need (monitoring, incident response, SLA, capacity, or runbooks)
2. Follow the corresponding workflow below
3. Produce structured outputs: monitoring dashboards, incident reports, SLA scorecards, or runbooks

Examples

• User: "Set up pipeline monitoring" Agent: Runs Monitoring workflow, defines alert thresholds, creates dashboard with latency/throughput/error rate metrics

• User: "Handle a pipeline failure" Agent: Runs Incident Response workflow, classifies severity, executes runbook, produces post-incident report with root cause

• User: "Create an on-call runbook" Agent: Runs Runbook Creation workflow, documents troubleshooting steps, escalation paths, and verification checks

When to Use

• Running daily data platform health checks, pipeline triage, and SLA enforcement
• Leading incident response, post-incident reviews, and on-call/shift handoffs
• Managing vendor escalations, cost reviews, and operational process design
• Coaching ops engineers on runbooks, capacity, and alert hygiene

When NOT to Use

• Strategic data mesh/lakehouse architecture or governance policy → use data-architect
• Program roadmaps, quarterly planning, or governance committee operations → use data-manager
• Writing dimensional models or MERGE-based incremental ETL → use data-warehouse-engineer
• General cloud/Kubernetes infrastructure outside the data platform → use infrastructure-engineer
• Org-wide on-call program, SEV definitions, postmortem tooling → use incident-management-engineer

Core Workflows

1. Platform Operations Oversight

Daily operational cadence:

Health check checklist:

• All critical pipelines completed successfully
• Data freshness within SLA thresholds
• No critical or high alerts active >30 min
• Storage utilization <85%
• Query performance within baseline
• Backup jobs completed

2. Incident & Problem Management

Incident severity matrix:

Incident lifecycle:

1. Detect → 2. Triage → 3. Mitigate → 4. Resolve → 5. Review

Post-incident review (within 48 hours for P1-P2):

• Timeline of events
• Root cause (5 Whys)
• Impact assessment
• Action items with owners and dates
• Process improvements

3. Team & Shift Leadership

On-call rotation design:

• Primary + secondary (overlapping coverage)
• Weekly rotations (not daily — too disruptive)
• Include weekend coverage in planning
• Escalation path: Engineer → Lead → Manager → Director

Shift handoff template:

## Shift Handoff — [Date] [Shift]

### Active Incidents
| ID | Severity | Status | Owner | Notes |
|---|---|---|---|---|
| INC-001 | P2 | Mitigated | @alice | Awaiting permanent fix |

### Alerts Requiring Attention
- [ ] Storage forecast will hit 90% in 3 days

### Changes Deployed
- [ ] Pipeline X updated to v2.1 (stable)

### Planned Work Next Shift
- [ ] Apply security patches to warehouse

### Issues for Lead Attention
- Recurring alert on pipeline Y — may need threshold tuning

4. Vendor & Cost Management

Monthly cost review:

• Actual vs budgeted spend
• Cost per TB processed, per pipeline run
• Identify optimization opportunities
• Vendor contract renewal timeline

Vendor escalation path:

1. Technical support (standard ticket)
2. Account manager (business impact)
3. Executive escalation (contract-level)