Jaw Dev Devops

Dev-DevOps — Production Infrastructure & Delivery

Build reliable, secure, and automated infrastructure and delivery pipelines. This skill has modular references for specialized guidance — read the relevant ones before coding.

C0/C1 work (small local patches): See dev §0.0 Work Classifier + §0.1 Patch Fast-Path before reading references.

dev is canonical: dev §0.2 Rule Classes, §3 Verification Gate, and §5 Safety Rules apply to all work governed by this skill.

Severity mapping: CRITICAL/HIGH ⇒ STRICT; MEDIUM ⇒ DEFAULT (aligned with dev §0.2).

Modular References

Read package-release.md for package publishing, registry auth, npm/PyPI trusted publishing, Bun-to-npm release decisions, and downstream package channels. Read cross-platform-release.md when a release claim depends on OS-local behavior that CI may not prove. Read homebrew.md for Formula/Cask distribution work. Read platform-engineering.md for broader DevOps capability refresh, DORA, provider routing, and platform guardrails. Read docker.md + ci-cd-deploy.md first for containerized deploy workflows. For K8s-specific work, add kubernetes.md. For SRE/on-call, add sre-foundations.md.

When release, registry-auth, provider-doc, service-status, image/platform version, or package-manager behavior depends on current external evidence, read the active search skill and follow its source-fetch and evidence-status rules instead of relying on stale memory or copied snippets.

§1 Container Builds

§1.1 Dockerfile Rules (STRICT)

For canonical Dockerfile templates, read references/docker.md §1.

§1.2 Image Security & Supply-Chain Baseline (STRICT)

CRITICAL/HIGH findings → block push. No exceptions. The 2026 baseline is ONE workflow, not separate tips: minimal base image (§1.1) → SBOM generation (Syft / Docker Scout) → vulnerability scan (Trivy or Grype) → sign + attest (Cosign/Sigstore) → digest-based promote (§2.4). Read references/docker.md §4 for scan/SBOM/sign command examples, and ../jaw-dev-security/references/supply-chain-sbom.md for deeper SBOM/signing policy.

§1.3 Anti-Patterns

§2 Deploy Pipeline

§2.1 Pipeline Stages (DEFAULT)

[dev-testing §5]  lint → typecheck → test → contract → e2e
[dev-devops]      build-image → scan → push-registry → deploy-staging → smoke → promote → deploy-prod

§2.2 GHA Reusable Workflows (DEFAULT)

# .github/workflows/ci.yml (caller)
jobs:
  build:
    uses: org/templates/.github/workflows/build-test.yml@v2
    with:
      service: payments
    secrets: inherit

§2.3 Deploy Strategies (DEFAULT)

§2.4 Rollback Rules (STRICT)

• Every deployment must be rollback-capable within 5 minutes
• Digest-based promote only — mutable tags are banned
• DB migrations: forward-only + backward-compatible (expand-contract pattern)
• Post-rollback: automatic Slack/PagerDuty notification

§2.5 Secret Management (STRICT)

Rule (DEVOPS-AUTH-01): prefer OIDC, workload identity federation, trusted publishing, or another short-lived credential flow before static long-lived tokens. When a static token is unavoidable, scope it narrowly, store it in the managed secret system, and rotate on schedule or on incident.

§2.6 GitOps (DEFAULT)

• Actions = CI, ArgoCD = CD — separation of concerns
• Actions updates deploy repo (image digest PR/commit) → ArgoCD reconciles
• Self-heal: ArgoCD auto-reverts drift
• Environment protection: GitHub Environments for prod approval gate

§2.7 Release Proof Contract (STRICT)

Rule (DEVOPS-RELEASE-PROOF-01): a release claim must name the artifact digest, the workflow or builder identity, the deploy target and environment, smoke-test evidence, and rollback evidence. Five items, each an artifact rather than an assertion — "deployed successfully" names none of them.

Keep the proof at this level; detailed package, platform, and supply-chain mechanics live in the release references.

§2.8 Freeze and GO/NO-GO Gates (STRICT)

DEVOPS-RELEASE-PROOF-01 governs the proof bundle for an artifact you already published. This section governs the decision to publish at all — the readiness report, and the gates it claims to have passed.

Why gate-weakening is the load-bearing one. In the case that produced these rules, a freeze audit rejected the first GO report on three counts: unresolved review threads on merged PRs, a red gate argued into an exception, and missing frozen-head receipts. The red-gate part is the instructive one — the suite was red, the red tests were known to be load-sensitive, and the fix was real, so the report explained the exception. The audit rejected that anyway, and correctly: an exception argued after a gate fails is indistinguishable from an exception argued because it failed.

The honest move was to decompose the gate to match what CI actually runs (DEVOPS-SUITE-PARTITION-01), which turned the general suite green. The one remaining local failure was waived separately on the DEVOPS-BASELINE-DEFECT-01 triple — not by the partitioning. Two different mechanisms for two different problems; collapsing them is how a partition becomes a way to hide a defect.

Operational mechanics — suite partitioning, baseline-versus-defect attribution, instrument stability, and exact-head evidence — live in references/ci-cd-deploy.md §6. Runtime and operator-signal evidence rules live in references/sre-foundations.md §7.

§2.9 Branch Lifecycle Hygiene (STRICT)

Delivery repositories accumulate dead refs, and the cost is not disk. Stale branches make the remote branch list unusable for triage, keep superseded heads reachable by tooling that resolves names, and hide the handful of branches that actually still matter. Treat branch lifecycle as delivery infrastructure.

Why the merged/closed distinction is load-bearing. delete_branch_on_merge reads as complete branch hygiene, so a repository with it enabled looks solved. It is not: in the case behind this rule the setting was on and the repository still carried 59 dead remote branches, because closed-unmerged PRs fall outside what it covers. The failure is silent and compounds — nothing reports it, and the branch list simply degrades until triage stops using it.

Why stacked PRs break naive cleanup. A stacked child PR targets its parent's head branch, so deleting a closed parent closes the open child. "The PR that owned this branch is closed" is therefore insufficient grounds for deletion: the base of any open PR is protected regardless of its own PR state (jaw-dev DEV-STACK-04).

Fork heads are out of scope, and identity is by repository id. A fork's head lives in the contributor's repository; deleting refs there is neither permitted nor intended. Compare repository ids, not names — a fork commonly carries the same branch names as upstream, so name comparison silently misclassifies it.

Mechanics, the deletion-plan algorithm, and a worked audit live in references/branch-lifecycle.md.

§3 Kubernetes Basics

§3.1 Minimum Viable K8s (DEFAULT)

§3.2 Gateway API (v1.6+, verified 2026-07-02)

Gateway API is the successor for new routing while Ingress remains GA but feature-frozen (not planned for removal). v1.6.0 (2026-06) graduates TCPRoute and UDPRoute to GA. Role separation: platform team owns GatewayClass + Gateway, app team owns HTTPRoute. Progressive delivery pairs directly with it: Flagger supports Gateway API HTTPRoute canaries.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payments-route
spec:
  parentRefs:
    - name: shared-gateway
  hostnames: ["payments.example.com"]
  rules:
    - matches:
        - path: { type: PathPrefix, value: /api }
      backendRefs:
        - name: payments-svc
          port: 8080

§3.3 Scaling (DEFAULT)

§3.4 Anti-Patterns

§4 Infrastructure as Code

§4.1 OpenTofu/Terraform Rules (DEFAULT)

§4.2 Tool Selection (HEURISTIC)

§4.3 Anti-Patterns

§5 SRE Foundations

§5.1 SLO/SLI (DEFAULT)

SLIs measure user experience, not infrastructure metrics. "CPU is fine ≠ users are fine."

Error budget = 1 − SLO (99.9% → 0.1% budget).

DORA 2025 (verified 2026-07-02): AI acts as an amplifier — returns depend on the underlying sociotechnical system. For AI-agent-heavy delivery, invest in golden paths, guardrails, observability, provenance, and review gates, not just faster merges.

§5.2 Error Budget Policy (DEFAULT)

Single incident consuming >20% → mandatory postmortem. Two consecutive window misses → architecture review.

§5.3 Incident Response (DEFAULT)

1. Detect → Triage (S1/S2/S3) → Stabilize → Fix → Postmortem
2. Roles: IC, Primary Responder, Comms Lead, Scribe
3. Mitigation first, diagnosis second during active incidents
4. All S1/S2 → mandatory blameless postmortem within 5 business days
5. Status updates: S1 every 15min, S2 every 30min

§5.4 Runbook Template (HEURISTIC)

## [Service] — [Symptom]
### Diagnosis
1. Logs: `kubectl logs -l app=<name> --tail=100`
2. Metrics: Grafana → [dashboard URL]
3. Dependencies: `curl -s http://<dep>/health | jq .`
### Emergency Mitigation
1. Rollback: `argocd app rollback <app>`
2. Traffic block: ...
### Root Fix
1. ...
### Escalation
- Owner: @team-sre
- PagerDuty: [policy]

§5.5 Anti-Patterns

§6 Cross-References

dev-devops owns: container builds, deploy pipelines, K8s manifests, IaC modules, SRE/incident response, edge infra, ML infra. DevOps owns operational scan execution and release gates; dev-security owns security policy, severity thresholds, and required evidence. dev-backend owns: application-layer observability code, API design, health check implementation. Overlap: observability alerting rules (dev-devops §5) ↔ observability code instrumentation (dev-backend observability.md). Cross-ref both.

Pre-flight Checklist

Before submitting infrastructure changes:

• Dockerfile is multi-stage with distroless/slim final image and non-root user
• Image scanned (Trivy/Scout) with CRITICAL/HIGH gate — no unresolved findings
• SBOM generated and attestation signed (Cosign) for production images
• Deploy pipeline uses digest-based promotion, never mutable tags
• K8s manifests have resource requests/limits, probes, PDB, and use Gateway API (not Ingress)
• IaC uses remote state, pinned versions, and modular decomposition
• Secrets are managed through Vault/AWS SM/GHA Secrets — no .env commits, no ARG secrets
• SLO/SLI defined with error budget policy and burn-rate alerting
• Rollback plan documented and tested — <5min rollback capability confirmed
• Runbook exists for critical failure scenarios