Overview
Reversibility and measurability are the two non-negotiables: any production change must roll back cleanly, and any SLO breach must surface as a metric and an alert. Delivery runs on four rails: an automated infrastructure-as-code foundation, CI/CD pipelines with rollback guarantees, zero-downtime database and application deployments, and deep observability telemetry (metrics, logs, traces, and SLO alerts).
[Phase 1: IaC & Environment Topology] ──► [Phase 2: Build & Packaging] ──► [Phase 3: CI/CD Pipeline Gates]
│
[Phase 5: Observability & SRE] ◄── [Phase 4: Zero-Downtime Rollout] ◄────┘
The DevOps and SRE engineer ensures infrastructure reproducibility, eliminates manual drift, and protects system availability through automated health gates and rollback automation.
When to Use
- When designing, provisioning, or modifying cloud infrastructure with Terraform, OpenTofu, or Ansible.
- When creating, hardening, or optimizing Docker container images and multi-stage build workflows.
- When configuring automated CI/CD pipelines (GitHub Actions, GitLab CI) with lint, test, and security gates.
- When executing zero-downtime database schema migrations or blue-green/canary application deployments.
- When deploying Kubernetes manifests, Helm charts, and ingress routing controllers.
- When implementing Prometheus metrics, Grafana dashboards, structured logging, and SLO alert thresholds.
When NOT to Use
- When writing business domain application logic or database entity models (use
development/role-software-engineer).
- When conducting penetration testing or dynamic security exploitation sweeps (use
cybersecurity/role-security-engineer).
- When profiling and tuning complex SQL database query plans (use
data/role-data-engineer).
- When designing UI/UX frontend styling and client components (use
content/role-ui-ux-designer).
Deterministic Dispatch Matrix
| Operational Goal / Infrastructure Challenge |
Delegated Child Skill |
Primary Operational Artifact |
| Provisioning and state management for cloud resources (AWS/GCP/Azure) |
devops/terraform-iac |
Declarative Terraform modules, plan audits, drift mitigation |
| Container image packaging, layer caching, and multi-stage optimization |
devops/docker |
Hardened, minimal non-root Dockerfiles with build cache |
| Automating test-and-deploy pipelines with status checks and gates |
devops/ci-cd-pipeline |
GitHub Actions / GitLab CI workflows with rollback triggers |
| Non-blocking DDL migrations, Expand/Contract schema evolution |
development/zero-downtime-migration-audit |
Non-locking migration scripts (CONCURRENTLY, dual-writing) |
| Kubernetes packaging, parameterization, and cluster deployment |
devops/helm-charts |
Versioned Helm charts, templated values, release manifests |
| Server configuration management, OS hardening, fleet orchestration |
devops/ansible-automation |
Idempotent Ansible playbooks and inventory configurations |
| Distributed tracing, metric collection, dashboards, and SLO alerting |
devops/observability |
Prometheus alert rules, Grafana dashboards, OpenTelemetry config |
Workflow
Phase 1: Infrastructure as Code & Topology Modeling
- Model all cloud infrastructure declaratively using
devops/terraform-iac.
- Ensure state files are stored in remote encrypted backends with distributed state locking.
- Execute
terraform plan and audit for unintended destructive resource replacements before applying.
Phase 2: Hermetic Container Packaging
- Package services using multi-stage Docker builds via
devops/docker.
- Enforce non-root user execution, pin dependency versions, and minimize attack surface.
- Eliminate secrets and build artifacts from final runtime images.
Phase 3: CI/CD Pipeline & Quality Gates
- Define hermetic build, test, and release pipelines using
devops/ci-cd-pipeline.
- Scaffold automated quality gates: type checking, unit tests, security scans, and image vulnerability audits.
- Configure immutable release tags and verifiable artifact digests.
Phase 4: Zero-Downtime Deployment & Canary Rollout
- For database modifications, dispatch to
development/zero-downtime-migration-audit to execute the Expand/Contract pattern.
- Deploy applications to Kubernetes clusters using parameterized charts via
devops/helm-charts.
- Configure blue-green or canary release strategies with automated health probes (
livenessProbe, readinessProbe).
- Enforce automated rollback triggers if error rates spike during rollout.
Phase 5: Observability, SRE & Fleet Hardening
- Scaffold metric scraping, log aggregation, and trace sampling using
devops/observability.
- Define Service Level Objectives (SLOs) and Error Budgets for latency, availability, and throughput.
- Configure automated fleet configuration and security patching with
devops/ansible-automation.
Anti-Rationalization Table
| DevOps Excuse |
Flawed Rationalization |
Required SRE Discipline |
| "I will configure the server manually via SSH" |
"It's faster than writing an Ansible playbook or Terraform." |
Manual changes create snowflake servers and untracked drift. Every change must be codified in IaC. |
"Deploying with latest tag is fine" |
"It always pulls the most recent build automatically." |
Mutable tags cause non-deterministic deployments and break rollbacks. Pin explicit commit SHAs or semantic tags. |
| "We don't need health checks for internal services" |
"If the pod crashes, Kubernetes restarts it anyway." |
Crash loops without readiness probes route traffic to broken pods. Configure explicit liveness and readiness endpoints. |
"Run a quick ALTER TABLE during off-peak hours" |
"It will only lock the table for a few seconds." |
Table locks during peak spikes cause cascading connection pool exhaustion. Use non-blocking DDL and Expand/Contract. |
| "Alert on every CPU spike over 80%" |
"We want to know whenever the server is working hard." |
Non-actionable alerts cause alert fatigue. Alert strictly on user-impacting symptoms (latency, 5xx errors, breach of SLO). |
Implementation Examples
Infrastructure Deployment & Canary Health Gate (Bash)
#!/usr/bin/env bash
set -euo pipefail
NAMESPACE="production"
DEPLOYMENT="payment-service"
CANARY_URL="https://canary.example.com/healthz"
echo "=== Step 1: Deploying Helm Canary Release ==="
helm upgrade --install "$DEPLOYMENT-canary" ./charts/payment \
--namespace "$NAMESPACE" \
--set replicaCount=1 \
--set canary.enabled=true \
--wait --timeout=120s
echo "=== Step 2: Validating Canary Health Probes ==="
STATUS_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$CANARY_URL" || echo "000")
if [ "$STATUS_CODE" -ne 200 ]; then
echo "ERROR: Canary health check failed with HTTP $STATUS_CODE! Triggering rollback..."
helm rollback "$DEPLOYMENT-canary" --namespace "$NAMESPACE"
exit 1
fi
echo "SUCCESS: Canary healthy. Proceeding with full cluster rollout."
helm upgrade --install "$DEPLOYMENT" ./charts/payment \
--namespace "$NAMESPACE" \
--wait --timeout=300s
Automated Terraform Drift Detector (Python)
import subprocess
import json
import sys
def check_terraform_drift(working_dir: str = ".") -> bool:
"""Executes terraform plan and verifies zero unplanned resource drift."""
print(f"Checking Terraform drift in {working_dir}...")
cmd = ["terraform", "plan", "-detailed-exitcode", "-no-color"]
res = subprocess.run(cmd, cwd=working_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
# Detailed exit codes: 0 = No changes, 1 = Error, 2 = Succeeded with changes present
if res.returncode == 0:
print("SUCCESS: Infrastructure is clean. Zero drift detected.")
return True
elif res.returncode == 2:
print("WARNING: Infrastructure drift detected! Changes are pending:")
print(res.stdout)
return False
else:
print(f"ERROR: Terraform plan failed!\n{res.stderr}")
sys.exit(1)
if __name__ == "__main__":
print("Terraform drift detector initialized.")
Verification Checklist
npm install -g cli-tool
1---2name: role-devops-sre-engineer3description: Use when orchestrating infrastructure as code, CI/CD pipelines, container4 orchestration, zero-downtime deployments, observability metrics, and site reliability5 engineering.6domain: devops7author: oyi778version: 1.0.09category: devops10tags:11- devops12- sre13- infrastructure14- docker15- terraform16license: MIT17subdomain: devops-sre18triggers:19- devops role20- sre role21- zero downtime deploy22- terraform module23- ci cd pipeline24- slo error budget25- observability metrics26- kubernetes deployment27metadata:28 source: https://skillmd.com/skills/oyi77/role-devops-sre-engineer29---303132## Overview3334Reversibility and measurability are the two non-negotiables: any production change must roll back cleanly, and any SLO breach must surface as a metric and an alert. Delivery runs on four rails: an automated infrastructure-as-code foundation, CI/CD pipelines with rollback guarantees, zero-downtime database and application deployments, and deep observability telemetry (metrics, logs, traces, and SLO alerts).3536```37[Phase 1: IaC & Environment Topology] ──► [Phase 2: Build & Packaging] ──► [Phase 3: CI/CD Pipeline Gates]38 │39 [Phase 5: Observability & SRE] ◄── [Phase 4: Zero-Downtime Rollout] ◄────┘40```4142The DevOps and SRE engineer ensures infrastructure reproducibility, eliminates manual drift, and protects system availability through automated health gates and rollback automation.4344## When to Use4546- When designing, provisioning, or modifying cloud infrastructure with Terraform, OpenTofu, or Ansible.47- When creating, hardening, or optimizing Docker container images and multi-stage build workflows.48- When configuring automated CI/CD pipelines (GitHub Actions, GitLab CI) with lint, test, and security gates.49- When executing zero-downtime database schema migrations or blue-green/canary application deployments.50- When deploying Kubernetes manifests, Helm charts, and ingress routing controllers.51- When implementing Prometheus metrics, Grafana dashboards, structured logging, and SLO alert thresholds.5253## When NOT to Use5455- When writing business domain application logic or database entity models (use `development/role-software-engineer`).56- When conducting penetration testing or dynamic security exploitation sweeps (use `cybersecurity/role-security-engineer`).57- When profiling and tuning complex SQL database query plans (use `data/role-data-engineer`).58- When designing UI/UX frontend styling and client components (use `content/role-ui-ux-designer`).5960## Deterministic Dispatch Matrix6162| Operational Goal / Infrastructure Challenge | Delegated Child Skill | Primary Operational Artifact |63|---|---|---|64| Provisioning and state management for cloud resources (AWS/GCP/Azure) | `devops/terraform-iac` | Declarative Terraform modules, plan audits, drift mitigation |65| Container image packaging, layer caching, and multi-stage optimization | `devops/docker` | Hardened, minimal non-root Dockerfiles with build cache |66| Automating test-and-deploy pipelines with status checks and gates | `devops/ci-cd-pipeline` | GitHub Actions / GitLab CI workflows with rollback triggers |67| Non-blocking DDL migrations, Expand/Contract schema evolution | `development/zero-downtime-migration-audit` | Non-locking migration scripts (`CONCURRENTLY`, dual-writing) |68| Kubernetes packaging, parameterization, and cluster deployment | `devops/helm-charts` | Versioned Helm charts, templated values, release manifests |69| Server configuration management, OS hardening, fleet orchestration | `devops/ansible-automation` | Idempotent Ansible playbooks and inventory configurations |70| Distributed tracing, metric collection, dashboards, and SLO alerting | `devops/observability` | Prometheus alert rules, Grafana dashboards, OpenTelemetry config |7172## Workflow7374### Phase 1: Infrastructure as Code & Topology Modeling75761. Model all cloud infrastructure declaratively using `devops/terraform-iac`.772. Ensure state files are stored in remote encrypted backends with distributed state locking.783. Execute `terraform plan` and audit for unintended destructive resource replacements before applying.7980### Phase 2: Hermetic Container Packaging81821. Package services using multi-stage Docker builds via `devops/docker`.832. Enforce non-root user execution, pin dependency versions, and minimize attack surface.843. Eliminate secrets and build artifacts from final runtime images.8586### Phase 3: CI/CD Pipeline & Quality Gates87881. Define hermetic build, test, and release pipelines using `devops/ci-cd-pipeline`.892. Scaffold automated quality gates: type checking, unit tests, security scans, and image vulnerability audits.903. Configure immutable release tags and verifiable artifact digests.9192### Phase 4: Zero-Downtime Deployment & Canary Rollout93941. For database modifications, dispatch to `development/zero-downtime-migration-audit` to execute the Expand/Contract pattern.952. Deploy applications to Kubernetes clusters using parameterized charts via `devops/helm-charts`.963. Configure blue-green or canary release strategies with automated health probes (`livenessProbe`, `readinessProbe`).974. Enforce automated rollback triggers if error rates spike during rollout.9899### Phase 5: Observability, SRE & Fleet Hardening1001011. Scaffold metric scraping, log aggregation, and trace sampling using `devops/observability`.1022. Define Service Level Objectives (SLOs) and Error Budgets for latency, availability, and throughput.1033. Configure automated fleet configuration and security patching with `devops/ansible-automation`.104105## Anti-Rationalization Table106107| DevOps Excuse | Flawed Rationalization | Required SRE Discipline |108|---|---|---|109| "I will configure the server manually via SSH" | "It's faster than writing an Ansible playbook or Terraform." | Manual changes create snowflake servers and untracked drift. Every change must be codified in IaC. |110| "Deploying with `latest` tag is fine" | "It always pulls the most recent build automatically." | Mutable tags cause non-deterministic deployments and break rollbacks. Pin explicit commit SHAs or semantic tags. |111| "We don't need health checks for internal services" | "If the pod crashes, Kubernetes restarts it anyway." | Crash loops without readiness probes route traffic to broken pods. Configure explicit liveness and readiness endpoints. |112| "Run a quick `ALTER TABLE` during off-peak hours" | "It will only lock the table for a few seconds." | Table locks during peak spikes cause cascading connection pool exhaustion. Use non-blocking DDL and Expand/Contract. |113| "Alert on every CPU spike over 80%" | "We want to know whenever the server is working hard." | Non-actionable alerts cause alert fatigue. Alert strictly on user-impacting symptoms (latency, 5xx errors, breach of SLO). |114115## Implementation Examples116117### Infrastructure Deployment & Canary Health Gate (Bash)118119```bash120#!/usr/bin/env bash121set -euo pipefail122123NAMESPACE="production"124DEPLOYMENT="payment-service"125CANARY_URL="https://canary.example.com/healthz"126127echo "=== Step 1: Deploying Helm Canary Release ==="128helm upgrade --install "$DEPLOYMENT-canary" ./charts/payment \129 --namespace "$NAMESPACE" \130 --set replicaCount=1 \131 --set canary.enabled=true \132 --wait --timeout=120s133134echo "=== Step 2: Validating Canary Health Probes ==="135STATUS_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$CANARY_URL" || echo "000")136137if [ "$STATUS_CODE" -ne 200 ]; then138 echo "ERROR: Canary health check failed with HTTP $STATUS_CODE! Triggering rollback..."139 helm rollback "$DEPLOYMENT-canary" --namespace "$NAMESPACE"140 exit 1141fi142143echo "SUCCESS: Canary healthy. Proceeding with full cluster rollout."144helm upgrade --install "$DEPLOYMENT" ./charts/payment \145 --namespace "$NAMESPACE" \146 --wait --timeout=300s147```148149### Automated Terraform Drift Detector (Python)150151```python152import subprocess153import json154import sys155156def check_terraform_drift(working_dir: str = ".") -> bool:157 """Executes terraform plan and verifies zero unplanned resource drift."""158 print(f"Checking Terraform drift in {working_dir}...")159 160 cmd = ["terraform", "plan", "-detailed-exitcode", "-no-color"]161 res = subprocess.run(cmd, cwd=working_dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)162 163 # Detailed exit codes: 0 = No changes, 1 = Error, 2 = Succeeded with changes present164 if res.returncode == 0:165 print("SUCCESS: Infrastructure is clean. Zero drift detected.")166 return True167 elif res.returncode == 2:168 print("WARNING: Infrastructure drift detected! Changes are pending:")169 print(res.stdout)170 return False171 else:172 print(f"ERROR: Terraform plan failed!\n{res.stderr}")173 sys.exit(1)174175if __name__ == "__main__":176 print("Terraform drift detector initialized.")177```178179## Verification Checklist180181- [ ] All infrastructure defined in version-controlled declarative IaC modules.182- [ ] Container images built using multi-stage non-root profiles with pinned versions.183- [ ] CI/CD pipeline enforces automated lint, test, and security verification gates.184- [ ] Zero-downtime database migrations validated with non-blocking DDL statements.185- [ ] Health and readiness probes configured for automated container traffic routing.186- [ ] Observability metrics, structured logging, and SLO alert triggers established.187188`npm install -g cli-tool`