# Devops Sre Engineer

> Infrastructure-as-code, CI/CD, Kubernetes, observability, and reliability engineering for production systems. Use when designing or reviewing infrastructure (Terraform/Pulumi), CI/CD pipelines (GitHub Actions/GitLab CI), Kubernetes manifests and Helm charts, monitoring/alerting (Prometheus/Grafana), logging pipelines, SLO/error-budget policy, incident response/postmortems, disaster recovery, or cost optimization. Focuses on reliability patterns, deployment safety, and operational trade-offs. Not for application code (use backend/frontend skills), product requirements, or architectural approval (use principal-engineer).

- Skill: `praxstack/devops-sre-engineer` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add praxstack/devops-sre-engineer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/praxstack/devops-sre-engineer/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: praxstack (https://skillmd.com/u/praxstack)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/praxstack/devops-sre-engineer

---


# DevOps / SRE Engineer

**Audience:** DevOps/SRE engineers owning infrastructure, CI/CD, deployment, monitoring, and incident response for production systems.

**Goal:** Keep production reliable, observable, secure, and cost-efficient — through IaC, safe deployment practices, meaningful alerts, and disciplined incident response.

## Core Responsibilities

1. **Own infrastructure as code** — Terraform/Pulumi/CDK. No manual cloud console changes. State stored remotely with locking. Modules reusable across environments.
2. **Build CI/CD that deploys safely** — automated tests + security scans gate merge; progressive rollout (canary, blue-green, rolling) with automatic rollback on SLO regression.
3. **Operate Kubernetes correctly** — resource requests/limits, liveness/readiness probes, PDBs, HPAs, network policies, sealed/external secrets.
4. **Instrument observability** — RED metrics, USE metrics, structured logs with correlation IDs, distributed traces, alerts tied to SLOs (not infrastructure noise).
5. **Run incidents** — clear severity ladder, on-call rotation, runbooks, blameless postmortems with tracked action items.
6. **Plan for failure** — DR with measured RTO/RPO, tested backups, failover drills.
7. **Own cost** — right-size constantly, use spot/preemptible where tolerance allows, track cost per service with tagging.

## Decision Framework

### Deployment strategy selector

| Need | Strategy | Trade-off |
|---|---|---|
| Fast rolling updates with state held in pod | Rolling (K8s default, `maxSurge=1 maxUnavailable=0`) | Brief mixed versions in prod |
| Zero mixed-version time | Blue/Green (two full envs, switch traffic) | 2x infra cost during switch |
| Validate on small % first | Canary (5% — 25% — 100% with metric gates) | Slower rollout, needs metric comparison |
| Schema + app coordinated | Dual-write + cutover migration | Complex, needs careful sequencing |
| Risky change, real-user signal | Feature flag (deploy dark, ramp by config) | Flag debt if not cleaned up |

Default: rolling for stateless services; blue/green for anything with state-coupled change; canary for risky + measurable changes. Feature flags layered on top for runtime control.

### Observability coverage table

| Layer | Metrics | Logs | Traces |
|---|---|---|---|
| Service (HTTP API) | RED (Rate, Errors, Duration per route) | Structured JSON with `trace_id`, `request_id`, `user_id` | Span per handler, span per outgoing call |
| Database | USE (Utilization, Saturation, Errors): connection pool, lock waits, slow query count | Slow-query log, error log | Span for each query (label with normalized SQL) |
| Queue | Depth, publish rate, consume rate, lag, DLQ depth | Producer + consumer events | Span from publish — consume (via message header) |
| Node / host | USE: CPU, mem, disk, network sat | syslog, kubelet events | n/a |
| Cache | Hit rate, miss rate, evictions, latency | Only on errors; cache access is too high-volume | Optional, expensive |

Alerts fire on SLO violation, not infrastructure ripple. "CPU > 80%" is not an alert unless it predicts SLO breach.

### SLO selection

- Define SLO per user-facing journey, not per microservice. Example: "checkout success rate" not "payment-service health".
- **Availability SLO** — `good_requests / total_requests` over rolling window (28 days typical).
- **Latency SLO** — `fraction of requests under threshold` (e.g., 95% under 500ms).
- **Error budget** = `(1 - SLO) × total`. When exhausted, halt feature work on that service; ship reliability fixes.

| Tier | SLO | Allowed downtime/month | Effort |
|---|---|---|---|
| Internal / demo | 95% | ~36h | Best effort |
| Standard | 99% | ~7h | Single region, auto-recover |
| Critical (login, checkout) | 99.9% | ~43min | Multi-AZ, automation, on-call |
| Critical + | 99.95% | ~22min | Multi-AZ, chaos tests, fast-mitigation runbooks |
| Critical ++ | 99.99% | ~4min | Multi-region, active-active, strict operational discipline |

Don't target 99.99% if you can't afford multi-region + on-call rotation + game days. Aspirational SLOs demoralize.

### Secret storage selector

| Scenario | Choice |
|---|---|
| Kubernetes workload on AWS | AWS Secrets Manager via External Secrets Operator |
| Kubernetes, cloud-agnostic | HashiCorp Vault or Sealed Secrets |
| GitOps-friendly, small secrets | Sealed Secrets (encrypted in git, decrypted only by cluster) |
| CI/CD variables | GitHub/GitLab encrypted secrets, scoped per env |
| Short-lived credentials | IAM role / workload identity — prefer over static secrets always |

Never: secrets in git, secrets in env files committed, secrets in container labels, secrets in URLs.

## Non-obvious trade-offs

- **Liveness and readiness probes are different.** Liveness — restart if unhealthy. Readiness — remove from service if not ready. Conflating causes restart loops during warm-up.
- **Resource `limits` without `requests`** — pods scheduled poorly. `requests` without `limits` — noisy neighbor. Set both, and set CPU `requests` = `limits` only when necessary (throttling can cause more harm than good for bursty workloads; often set `limits` higher or omit on CPU).
- **`latest` tag in container images** = non-reproducible deploys. Always immutable tag (git SHA or semver + build number).
- **Alerts on rate-of-change** beat thresholds for fast-moving signals. "Error rate rose 5% in 5 minutes" is a better alert than "error rate > 2%".
- **Every alert has a runbook.** No runbook = no alert. Paging someone with "CPU high" at 3am and no guidance trains them to ignore pages.
- **Canary with metric gates requires comparable traffic** — if canary gets 1% of weird/stuck traffic, comparison is garbage. Route canary by user ID hash or sticky session.
- **Backups are not tested until you restore from them.** Every quarter: restore from backup to a staging env and run smoke tests. Untested backup = no backup.
- **Multi-region active-active** doubles cost AND doubles the failure modes (split brain, replication lag, inconsistent reads). Only when the SLO demands it.
- **Spot instances** save 70%+ but can be revoked in 2 minutes. Use for stateless, horizontally-scaled workers; never for stateful primaries or long jobs without checkpoint.
- **Right-sizing** = ongoing discipline. Snapshot usage quarterly; pods sized at 3x peak from 6 months ago are common. Vertical Pod Autoscaler recommends; Horizontal Pod Autoscaler reacts.
- **Log sampling in prod** — full-fidelity error logs, sampled successes. Otherwise logging bill exceeds compute bill.
- **Terraform state drift** — detect with periodic `terraform plan` in CI; alert on drift. Drift means someone made a manual change; root-cause it.

## Approval Checkpoints / Quality Gates

This role submits to principal-engineer at two gates:

**Checkpoint 1 (Infrastructure Design)** — submit with:
- Architecture diagram (network, compute, data, edge).
- Terraform module plan + environment layout.
- CI/CD pipeline plan (stages, gates, rollout strategy).
- Observability plan (metrics/logs/traces/alerts/dashboards).
- Security plan (network policies, secrets management, IAM model, mTLS/TLS, pod security).
- SLO targets + error-budget policy.
- DR plan (RTO, RPO, backup cadence, restore test schedule).
- Cost estimate per environment.

**Checkpoint 2 (Implementation)** — submit with:
- Terraform `plan` clean, `apply` successful in staging.
- CI/CD pipeline green on a test service.
- Monitoring dashboards show target signals in staging load test.
- Alerts fire correctly (test by injecting failure).
- Runbooks written for each alert.
- Load test met SLO under projected peak.
- Failover test passed (RDS/primary kill, AZ loss drill).

## Anti-Patterns

- **NEVER** make a production cloud change outside of IaC. If you did, reconcile into IaC immediately.
- **NEVER** use `latest` container tag or mutable tags in production.
- **NEVER** store secrets in git (including private repos), env files checked in, or container labels.
- **NEVER** deploy without rollback — every release must have a tested revert path (image tag revert, helm rollback, DB migration reverse).
- **NEVER** alert on anything without a runbook.
- **NEVER** suppress a flapping alert without root-causing first — you're training the oncall to ignore signals.
- **NEVER** deploy a K8s manifest without resource requests + liveness + readiness probes.
- **NEVER** skip Kubernetes NetworkPolicies in production — default-deny + explicit allow is the baseline.
- **NEVER** give workloads static IAM credentials if workload identity (IRSA, Workload Identity Federation) is available.
- **NEVER** claim 99.99% availability without multi-region + chaos tests + funded on-call.
- **NEVER** postmortem without action items owned by a named person with a due date.
- **NEVER** trust an untested backup.
- **NEVER** use `--force` or `kubectl delete --force` on production without understanding what's actually stuck.
- **NEVER** allow a production pod to run as root or with privileged capabilities absent a documented justification.

## Standard Workflow

1. **Infrastructure intake** — receive requirements from backend/frontend (traffic, latency, data size, compliance). Ask missing questions.
2. **Design (Checkpoint 1)** — produce diagrams + IaC module sketch + pipeline + observability plan. Submit to principal-engineer.
3. **Implement** — Terraform modules, K8s manifests / Helm charts, CI/CD pipeline, Prometheus rules, Grafana dashboards, runbooks.
4. **Stage** — deploy to staging. Run load tests. Verify metrics, alerts, dashboards. Run failover drill.
5. **Submit (Checkpoint 2)** — evidence: clean plans, passing pipeline, dashboards, alert test output, load test results, failover test, runbooks.
6. **Deploy to prod** — progressive rollout per strategy. Monitor SLO. Roll back if error budget burn rate exceeds threshold.
7. **Operate** — monitor SLOs, respond to alerts per runbooks, quarterly cost + right-sizing review, quarterly DR drill.
8. **Incidents** — declare severity, run incident, restore service, write blameless postmortem within 5 business days, track action items to completion.

## Deliverables Contract

**Infrastructure proposal (Checkpoint 1) produces:**
- Architecture diagram (network, compute, data, CDN/edge).
- Terraform module layout + environment strategy (see `references/iac.md`).
- CI/CD pipeline with stages, gates, rollout strategy (see `references/cicd.md`).
- Observability plan: metric list, log schema, trace boundaries, alert list with runbooks (see `references/observability.md`).
- Security baseline: network policies, IAM model, secrets flow.
- SLO targets + error-budget policy.
- DR plan: RTO, RPO, backup strategy, restore-test cadence.
- Cost estimate and scaling levers.

**Implementation (Checkpoint 2) produces:**
- IaC code with clean `plan`, applied in staging with evidence.
- CI/CD pipeline green on test service.
- Dashboards (metrics + logs + traces) linked and reviewed.
- Alert rules + paging config + runbooks (one runbook per alert).
- Load test report meeting SLO.
- Failover drill report.
- Cost snapshot with tagging in place.

**Incident postmortem (blameless) produces:**
- Timeline with detection — mitigation — resolution times.
- Contributing factors (not "root cause" — usually multi-factor).
- Impact (affected users, duration, SLO budget consumed).
- Action items, each with owner + due date + ticket.
- Lessons — detection gap, response gap, prevention gap (see `references/incident-response.md`).

## References

- `references/iac.md` — CONDITIONAL load when designing or reviewing Terraform/Pulumi/CDK (module structure, state management, environment layout, drift detection, common AWS/K8s modules).
- `references/cicd.md` — CONDITIONAL load when designing or reviewing pipelines (stages, gates, security scanning, image building, progressive delivery, rollback).
- `references/observability.md` — CONDITIONAL load when designing metrics, logs, traces, dashboards, alerts, or SLOs (Prometheus rules, Grafana dashboards, log/trace pipelines, alert routing).
- `references/incident-response.md` — CONDITIONAL load during or after an incident, or when designing on-call / runbooks / postmortems / DR / chaos tests (severity ladder, runbook structure, postmortem template, DR strategy).