Dev-DevOps — Production Infrastructure & Delivery
Build reliable, secure, and automated infrastructure and delivery pipelines.
This skill has modular references for specialized guidance — read the relevant ones before coding.
C0/C1 work (small local patches): See jaw-dev §0.0 Work Classifier + §0.1 Patch Fast-Path before reading references.
jaw-dev is canonical: jaw-dev §0.2 Rule Classes, §3 Verification Gate, and §5 Safety Rules apply to all work governed by this skill.
Severity and rule authority are distinct (jaw-dev §0.2). Safety, correctness, and release proof remain mandatory; architecture and tool preferences need project-specific justification.
Modular References
| File |
When to Read |
What It Covers |
references/docker.md |
Container build/deploy |
Multi-stage builds, distroless, Docker Scout/Trivy, BuildKit secrets, SBOM/Cosign |
references/package-release.md |
Package publishing / release auth |
npm/PyPI trusted publishing, Bun-to-npm, registry auth model, downstream distribution table |
references/cross-platform-release.md |
Cross-platform release proof |
CI matrix vs local OS proof, Windows App/RDP prompts, desktop verification boundaries |
references/native-desktop-acceptance.md |
Electron desktop changes or release |
UI, packaged runtime, artifact identity, signing and distribution evidence |
references/macos-system-approvals.md |
macOS system prompts |
Human approval boundary, read-only checks, evidence for blocked rows |
references/repo-bootstrap.md |
Repository settings |
Read-back audit, rulesets, merge choices and PR limits |
references/agent-pr-intake.md |
Agent-contributed PRs |
Identity, draft policy and supersede proof |
references/local-gc.md |
Local worktree cleanup |
PR-state evidence, active/dirty guards and dry-run schedule |
../jaw-dev/references/stacked-prs.md |
Stacked PRs |
Canonical layer and CI rules; opt in to native stack tooling only where supported |
references/homebrew.md |
Homebrew distribution |
Formula vs Cask, audit/test, livecheck, artifact trust, install/uninstall proof |
references/platform-engineering.md |
Platform / DORA / provider routing |
DORA capabilities, platform guardrails, provider table rows, SLSA handoff |
references/kubernetes.md |
K8s deployment |
Gateway API (v1.6+), Kustomize overlays, HPA/VPA, Helm, ArgoCD GitOps |
references/ci-cd-deploy.md |
Deploy pipeline, and freeze/GO evidence |
GHA reusable workflows, deploy strategies, rollback, GitOps, progressive delivery; §6 suite partitioning, baseline-vs-defect, instrument stability, exact-head evidence, stability counting |
references/branch-lifecycle.md |
Branch or worktree cleanup |
Deletion-evidence checks, ref snapshot and restore, merged-vs-closed gap, stacked-PR bases, fork identity by repository id, worktree removal order |
references/iac.md |
Infrastructure code |
OpenTofu/Terraform modules, Pulumi, state encryption, blast radius isolation |
references/sre-foundations.md |
Operations/incidents |
SLO/SLI/error budget, burn-rate alerting, incident response, blameless postmortem |
references/edge-serverless.md |
Edge/serverless work |
Edge request shaping, auth at edge, Cloudflare Workers, Vercel Edge, edge AI triage |
references/ml-infra.md |
ML infrastructure |
GPU cluster mgmt, model registry, scaling, edge inference, MLOps platform patterns |
Read package-release.md for package publishing, registry auth, npm/PyPI
trusted publishing, Bun-to-npm release decisions, and downstream package
channels. Read cross-platform-release.md when a release claim depends on
OS-local behavior that CI may not prove. Read homebrew.md for Formula/Cask
distribution work. Read platform-engineering.md for broader DevOps capability
refresh, DORA, provider routing, and platform guardrails. Read docker.md + ci-cd-deploy.md first for containerized deploy workflows.
For K8s-specific work, add kubernetes.md. For SRE/on-call, add sre-foundations.md.
When release, registry-auth, provider-doc, service-status, image/platform
version, or package-manager behavior depends on current external evidence, read
the active search skill and follow its source-fetch and evidence-status rules
instead of relying on stale memory or copied snippets.
§1 Container Builds
§1.1 Dockerfile Rules (STRICT)
| Rule |
Detail |
| Multi-stage |
Separate build and runtime stages; final image has no build tools |
| Base image |
Pin version + SHA256 digest: node:22-slim@sha256:abc... |
| Minimal runtime base |
Prefer gcr.io/distroless/* (Debian-based, keyless-signed) — no shell/package manager; OR Chainguard/Wolfi images when nightly rebuilds, SBOM attestations, patch SLAs, or compliance matter |
| Non-root |
USER nonroot:nonroot (distroless) or create dedicated user |
| Dependency-first copy |
COPY package.json bun.lock ./ → install → COPY . . for layer caching |
| BuildKit secrets |
RUN --mount=type=secret,id=token ... — never use ARG for secrets |
.dockerignore |
.git, node_modules, .env*, *.log, dist/, coverage/, __pycache__/ |
For canonical Dockerfile templates, read references/docker.md §1.
§1.2 Image Security & Supply-Chain Baseline (STRICT)
CRITICAL/HIGH image findings block push. A later checklist exception cannot waive this gate;
any policy change must be approved and declared before the verdict. Other checklist
exceptions follow their owning policy. The 2026 baseline is ONE workflow,
not separate tips: minimal base image (§1.1) → SBOM generation (Syft / Docker Scout) →
vulnerability scan (Trivy or Grype) → sign + attest (Cosign/Sigstore) → digest-based
promote (§2.4). Read references/docker.md §4 for scan/SBOM/sign command examples, and
../jaw-dev-security/references/supply-chain-sbom.md for deeper SBOM/signing policy.
§1.3 Anti-Patterns
| Banned |
Symptom |
Fix |
FROM node:latest |
Irreproducible builds |
Pinned version + digest |
USER root in final stage |
Attack surface |
Non-root user |
COPY . . as first instruction |
Cache invalidation on every change |
Dependency files first |
ARG SECRET=xxx |
Exposed in image history |
BuildKit --mount=type=secret |
| No scan before push |
Vulnerable images in prod |
Trivy/Scout CI gate |
apt-get install without cleanup |
Bloated image |
--no-install-recommends && rm -rf /var/lib/apt/lists/* |
§2 Deploy Pipeline
§2.1 Pipeline Stages (DEFAULT)
[jaw-dev-testing §5] lint → typecheck → test → contract → e2e
[jaw-dev-devops] build-image → scan → push-registry → deploy-staging → smoke → promote → deploy-prod
For stacked PRs, use ../jaw-dev/references/stacked-prs.md for layer shape
(DEV-STACK-03), membership (DEV-STACK-06), CI diagnosis (DEV-STACK-07),
and the native-tool opt-in (DEV-STACK-OPT-IN-01).
§2.2 GHA Reusable Workflows (DEFAULT)
# .github/workflows/ci.yml (caller)
jobs:
build:
uses: org/templates/.github/workflows/build-test.yml@v2
with:
service: payments
secrets: inherit
| Rule |
Detail |
workflow_call |
Central CI template, max 10-level nesting |
| Permissions |
Caller cannot escalate; downgrade only |
| Environment |
environment: production + required reviewers + prevent self-review |
| Promote |
Digest-based (image@sha256:...), never mutable tags |
§2.3 Deploy Strategies (DEFAULT)
| Strategy |
Tool |
When |
Risk |
| Rolling update |
K8s Deployment |
Stateless, low risk |
Low |
| Blue-green |
Argo Rollouts blueGreen: |
Instant rollback, no DB migration |
Medium |
| Canary |
Argo Rollouts canary: steps: |
Traffic % control, metric-based promote |
Medium |
| Progressive |
Flagger |
Auto analysis + rollback, A/B testing |
Medium-High |
| Feature flag |
LaunchDarkly / Unleash |
Code-level gradual rollout |
Low |
§2.4 Rollback Rules (STRICT)
- Every deployment must be rollback-capable within 5 minutes
- Digest-based promote only — mutable tags are banned
- DB migrations: forward-only + backward-compatible (expand-contract pattern)
- Post-rollback: automatic Slack/PagerDuty notification
§2.5 Secret Management (STRICT)
Rule (DEVOPS-AUTH-01): prefer OIDC, workload identity federation, trusted
publishing, or another short-lived credential flow before static long-lived
tokens. When a static token is unavoidable, scope it narrowly, store it in the
managed secret system, and rotate on schedule or on incident.
| Source |
Usage |
| GHA Secrets / Vault / AWS SM |
CI pipeline secrets |
| External Secrets Operator |
K8s → Vault/AWS SM sync |
.env files |
Never committed — generated in CI |
| Rotation |
90-day cycle or immediate on incident |
§2.6 GitOps (DEFAULT)
- Actions = CI, ArgoCD = CD — separation of concerns
- Actions updates deploy repo (image digest PR/commit) → ArgoCD reconciles
- Self-heal: ArgoCD auto-reverts drift
- Environment protection: GitHub Environments for prod approval gate
§2.7 Release Proof Contract (STRICT)
Rule (DEVOPS-RELEASE-PROOF-01): a release claim must name the artifact digest,
the workflow or builder identity, the deploy target and environment, smoke-test
evidence, and rollback evidence. Five items, each an artifact rather than an
assertion — "deployed successfully" names none of them.
Keep the proof at this level; detailed package, platform, and supply-chain mechanics
live in the release references.
§2.8 Freeze and GO/NO-GO Gates (STRICT)
DEVOPS-RELEASE-PROOF-01 governs the proof bundle for an artifact you already
published. This section governs the decision to publish at all — the readiness
report, and the gates it claims to have passed.
| Rule |
Severity |
Statement |
DEVOPS-FREEZE-SHA-01 |
STRICT |
Pin the readiness report to the code SHA its gates describe. If the head moved after the freeze, prove the delta is docs-only (git diff --name-only <freeze> <head>) and keep every gate receipt on the freeze SHA. |
DEVOPS-GATE-WEAKEN-01 |
STRICT |
A red named gate is never excused inside the report that gate failed. Make the original command green, or replace it with a pre-declared equivalent CI actually runs — and declare the swap before the verdict, not after the failure. |
DEVOPS-REVIEW-THREADS-01 |
STRICT |
Unresolved review threads on merged PRs are a GO blocker. Count them after merge: a thread opened minutes before merge still counts until it is fixed or explicitly dismissed. |
DEVOPS-GATE-OWNER-01 |
STRICT |
A mandatory GO gate needs an implementing work-phase and a recorded terminal outcome — pass, not-reproduced, or explicitly deregistered. A gate nobody implements is not a gate; it is a wish. |
Why gate-weakening is the load-bearing one. In the case that produced these
rules, a freeze audit rejected the first GO report on three counts: unresolved
review threads on merged PRs, a red gate argued into an exception, and missing
frozen-head receipts. The red-gate part is the instructive one — the suite was red,
the red tests were known to be load-sensitive, and the fix was real, so the report
explained the exception. The audit rejected that anyway, and correctly: an
exception argued after a gate fails is indistinguishable from an exception argued
because it failed.
The honest move was to decompose the gate to match what CI actually runs
(DEVOPS-SUITE-PARTITION-01), which turned the general suite green. The one
remaining local failure was waived separately on the DEVOPS-BASELINE-DEFECT-01
triple — not by the partitioning. Two different mechanisms for two different
problems; collapsing them is how a partition becomes a way to hide a defect.
Operational mechanics — suite partitioning, baseline-versus-defect attribution,
instrument stability, and exact-head evidence — live in
references/ci-cd-deploy.md §6. Runtime and operator-signal evidence rules live in
references/sre-foundations.md §7.
§2.9 Branch Lifecycle Hygiene (STRICT)
This guidance does not itself authorize repository-setting changes, scheduled jobs,
branch or worktree deletion. Apply the task's actual write authority.
Delivery repositories accumulate dead refs, and the cost is not disk. Stale branches
make the remote branch list unusable for triage, keep superseded heads reachable by
tooling that resolves names, and hide the handful of branches that actually still
matter. Treat branch lifecycle as delivery infrastructure.
| Rule |
Severity |
Statement |
DEVOPS-BRANCH-AUTODELETE-01 |
STRICT |
Enable host-side head deletion on merge (delete_branch_on_merge on GitHub) and close the gap it leaves. That setting fires only on merge; a pull request closed without merging keeps its head branch forever, so the closed-PR case needs its own scheduled automation. |
DEVOPS-BRANCH-DELETE-EVIDENCE-01 |
STRICT |
Never bulk-prune. Before deleting any ref, prove per branch that it is not protected, not an open PR head, not the base of an open PR, not a fork head, and not carrying unique commits. A name pattern is not evidence. |
DEVOPS-BRANCH-SNAPSHOT-01 |
STRICT |
Snapshot git for-each-ref (SHA + refname) for every local and remote ref to scratch space before the first deletion. A deleted remote branch is restorable with git push origin <sha>:refs/heads/<name> only while you still hold the SHA. |
DEVOPS-WORKTREE-DIRTY-01 |
STRICT |
Check every attached worktree for uncommitted work before removing it, and remove worktrees before their branches — an attached branch cannot be deleted, and --force on a dirty tree discards work no reflog will return. |
DEVOPS-BRANCH-NAMESPACE-01 |
STRICT |
Closed-PR cleanup needs a repository-declared disposable prefix and an exact match between current tip and a related closed PR head SHA; see branch-lifecycle.md. |
DEVOPS-REPO-BOOTSTRAP-01 |
DEFAULT |
Read back repository settings before proposing changes; see repo-bootstrap.md. |
DEVOPS-AGENT-INTAKE-01 |
DEFAULT |
Use the repository's declared PR review policy; see agent-pr-intake.md. |
DEVOPS-LOCAL-GC-01 |
STRICT |
Classify by PR state and SHA; snapshot refs, exclude active/dirty/locked or uncertain trees, and dry-run before cleanup; see local-gc.md. |
Why the merged/closed distinction is load-bearing. delete_branch_on_merge
reads as complete branch hygiene, so a repository with it enabled looks solved. It
is not: in the case behind this rule the setting was on and the repository still
carried 59 dead remote branches, because closed-unmerged PRs fall outside what it
covers. The failure is silent and compounds — nothing reports it, and the branch
list simply degrades until triage stops using it.
Why stacked PRs break naive cleanup. A stacked child PR targets its parent's
head branch, so deleting a closed parent closes the open child. "The PR that
owned this branch is closed" is therefore insufficient grounds for deletion: the
base of any open PR is protected regardless of its own PR state
(jaw-dev DEV-STACK-04).
Fork heads are out of scope, and identity is by repository id. A fork's head
lives in the contributor's repository; deleting refs there is neither permitted nor
intended. Compare repository ids, not names — a fork commonly carries the same
branch names as upstream, so name comparison silently misclassifies it.
Mechanics, the deletion-plan algorithm, and a worked audit live in
references/branch-lifecycle.md. Repository policy choices, contributed PR
review and local worktree cleanup live in references/repo-bootstrap.md,
references/agent-pr-intake.md and references/local-gc.md.
§3 Kubernetes Basics
§3.1 Minimum Viable K8s (DEFAULT)
| Resource |
Purpose |
| Deployment |
Pod template + replica management |
| Service |
Internal networking |
| HTTPRoute (Gateway API) |
External traffic routing — not Ingress |
| ResourceQuota |
Request/limit enforcement |
| Probes |
Liveness + readiness + startup |
| Namespace |
Environment isolation (dev/staging/prod) |
§3.2 Gateway API (v1.6+, verified 2026-07-02)
Gateway API is the successor for new routing while Ingress remains GA but feature-frozen
(not planned for removal). v1.6.0 (2026-06) graduates TCPRoute and UDPRoute to GA. Role
separation: platform team owns GatewayClass + Gateway, app team owns HTTPRoute.
Progressive delivery pairs directly with it: Flagger supports Gateway API HTTPRoute canaries.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: payments-route
spec:
parentRefs:
- name: shared-gateway
hostnames: ["payments.example.com"]
rules:
- matches:
- path: { type: PathPrefix, value: /api }
backendRefs:
- name: payments-svc
port: 8080
§3.3 Scaling (DEFAULT)
| Mechanism |
Scope |
Watch |
| HPA |
CPU/memory + custom metrics |
Don't combine with VPA on same metric |
| VPA |
Request auto-tuning |
Use Off mode for recommendations only |
| PDB |
Disruption budget |
minAvailable: 50% or maxUnavailable: 1 |
§3.4 Anti-Patterns
| Banned |
Fix |
Ingress (new projects) |
Gateway API HTTPRoute |
| No resource limits |
Always set requests + limits |
image: app:latest |
SHA digest or pinned SemVer |
| Single replica in prod |
Minimum 2 + PDB |
| Secrets in ConfigMap |
K8s Secret + External Secrets Operator |
| Annotation-based routing |
Gateway API native fields |
§4 Infrastructure as Code
§4.1 OpenTofu/Terraform Rules (DEFAULT)
| Rule |
Detail |
| State |
Remote backend required (S3+DynamoDB / TF Cloud / OpenTofu) |
| Encryption |
OpenTofu native state/plan encryption via KMS |
| Blast radius |
Separate state per app/layer/env |
| Modules |
Purpose-built (vpc, iam, ecs-service), typed I/O |
| Apply |
plan → PR review → apply; auto-apply staging only |
| Versions |
Pin provider + module versions explicitly |
§4.2 Tool Selection (HEURISTIC)
| Tool |
Best For |
Status (verified 2026-07-02) |
| OpenTofu (HCL) |
Open-source/licensing-neutral IaC default (MPL-2.0, Linux Foundation; v1.12.x) |
✅ Recommended for OSS neutrality |
| Terraform / HCP Terraform (BSL) |
Vendor support or HashiCorp platform integration required |
✅ Active (BSL license — check competitive-use terms) |
| Pulumi (TS/Python) |
Teams preferring programming languages |
✅ Active |
| AWS CDK |
AWS-only infrastructure |
✅ Active (AWS only) |
| CDKTF |
— |
❌ Deprecated 2025-12-10; repo archived/read-only, no further fixes |
§4.3 Anti-Patterns
| Banned |
Fix |
| Local state file |
Remote backend required |
| Manual console changes |
All changes via code |
| Hardcoded values |
Variables + tfvars |
| Monolithic main.tf |
Modular decomposition |
| CDKTF (new projects) |
OpenTofu or Pulumi |
| Unpinned provider versions |
Explicit version constraints |
§5 SRE Foundations
§5.1 SLO/SLI (DEFAULT)
| SLI |
Measurement |
Typical SLO |
| Availability |
Success requests / total |
99.9% (28-day rolling) |
| Latency |
p50/p95/p99 response time |
p99 < 500ms |
| Error rate |
5xx / total |
< 0.1% |
| Freshness |
Data update delay |
< 5min (pipelines) |
SLIs measure user experience, not infrastructure metrics. "CPU is fine ≠ users are fine."
Error budget = 1 − SLO (99.9% → 0.1% budget).
DORA 2025 (verified 2026-07-02): AI acts as an amplifier — returns depend on the
underlying sociotechnical system. For AI-agent-heavy delivery, invest in golden paths,
guardrails, observability, provenance, and review gates, not just faster merges.
§5.2 Error Budget Policy (DEFAULT)
| Budget State |
Action |
| Normal (>50%) |
Continue releases, routine monitoring |
| Accelerated burn (20–50%) |
Heightened alerts, slow releases, reliability triage |
| Exhausted (≤0%) |
Feature freeze — security/bugfix only; VP exception required |
Single incident consuming >20% → mandatory postmortem.
Two consecutive window misses → architecture review.
§5.3 Incident Response (DEFAULT)
- Detect → Triage (S1/S2/S3) → Stabilize → Fix → Postmortem
- Roles: IC, Primary Responder, Comms Lead, Scribe
- Mitigation first, diagnosis second during active incidents
- All S1/S2 → mandatory blameless postmortem within 5 business days
- Status updates: S1 every 15min, S2 every 30min
§5.4 Runbook Template (HEURISTIC)
## [Service] — [Symptom]
### Diagnosis
1. Logs: `kubectl logs -l app=<name> --tail=100`
2. Metrics: Grafana → [dashboard URL]
3. Dependencies: `curl -s http://<dep>/health | jq .`
### Emergency Mitigation
1. Rollback: `argocd app rollback <app>`
2. Traffic block: ...
### Root Fix
1. ...
### Escalation
- Owner: @team-sre
- PagerDuty: [policy]
§5.5 Anti-Patterns
| Banned |
Fix |
| Infrastructure-only SLIs |
User-experience-based SLIs |
| SLO without consequences |
Error budget policy with freeze gate |
| Too many SLIs (>5 per service) |
2-4 meaningful SLIs |
| Page on every deviation |
Burn-rate multi-window alerting |
| Blame individuals |
Blameless postmortem, system improvement |
| No error budget policy |
Define 3-stage policy (normal/accelerated/exhausted) |
§6 Cross-References
| Topic |
Canonical Owner |
What jaw-dev-devops defers |
| Test strategy & CI test stages |
jaw-dev-testing §5 |
Test pyramid, coverage gates |
| Backend observability code patterns |
jaw-dev-backend observability.md |
OTel SDK setup, structured logging |
| Security hardening (app-layer) |
jaw-dev-security |
OWASP, auth, input validation |
| SBOM/signing depth |
jaw-dev-security references/supply-chain-sbom.md |
Supply-chain evidence policy beyond image scan gates |
| Architecture module boundaries |
jaw-dev-architecture |
Coupling taxonomy, barrel discipline |
| Scaffolding conventions |
jaw-dev-scaffolding |
File naming, project structure |
| Frontend build/bundle |
jaw-dev-frontend |
Vite/webpack config, SSR |
jaw-dev-devops owns: container builds, deploy pipelines, K8s manifests, IaC modules, SRE/incident response, edge infra, ML infra. DevOps owns operational scan execution and release gates; jaw-dev-security owns security policy, severity thresholds, and required evidence.
jaw-dev-backend owns: application-layer observability code, API design, health check implementation.
Overlap: observability alerting rules (jaw-dev-devops §5) ↔ observability code instrumentation (jaw-dev-backend observability.md). Cross-ref both.
Pre-flight Checklist
Before submitting infrastructure changes:
1---2name: jaw-dev-devops3description: "MUST USE for infrastructure and delivery work — container builds, deploy pipelines, repository settings and cleanup, agent PR intake, Electron desktop acceptance, macOS approvals, Kubernetes, IaC, SRE, edge/serverless, ML infrastructure. Triggers: Dockerfile, K8s manifests, CI/CD pipeline, Terraform/IaC, release/deploy, repository rulesets, worktree cleanup, desktop packaging, devops/infra/deploy or release_cd task_tags."4metadata:5 source: https://skillmd.com/skills/lidge-jun/jaw-dev-devops6 short-description: "Infrastructure, release, repository hygiene, and desktop acceptance."7 keywords: "container, kubernetes, deploy, iac, sre, pipeline, supply chain, sbom, repository settings, worktree cleanup, agent pr, electron, macos approval"8 last-verified: "2026-10-08"9---1011# Dev-DevOps — Production Infrastructure & Delivery1213Build reliable, secure, and automated infrastructure and delivery pipelines.14This skill has modular references for specialized guidance — read the relevant ones before coding.1516> **C0/C1 work (small local patches):** See `jaw-dev` §0.0 Work Classifier + §0.1 Patch Fast-Path before reading references.1718> **`jaw-dev` is canonical:** `jaw-dev` §0.2 Rule Classes, §3 Verification Gate, and §5 Safety Rules apply to all work governed by this skill.1920Severity and rule authority are distinct (`jaw-dev` §0.2). Safety, correctness, and release proof remain mandatory; architecture and tool preferences need project-specific justification.2122## Modular References2324| File | When to Read | What It Covers |25|------|-------------|----------------|26| `references/docker.md` | Container build/deploy | Multi-stage builds, distroless, Docker Scout/Trivy, BuildKit secrets, SBOM/Cosign |27| `references/package-release.md` | Package publishing / release auth | npm/PyPI trusted publishing, Bun-to-npm, registry auth model, downstream distribution table |28| `references/cross-platform-release.md` | Cross-platform release proof | CI matrix vs local OS proof, Windows App/RDP prompts, desktop verification boundaries |29| `references/native-desktop-acceptance.md` | Electron desktop changes or release | UI, packaged runtime, artifact identity, signing and distribution evidence |30| `references/macos-system-approvals.md` | macOS system prompts | Human approval boundary, read-only checks, evidence for blocked rows |31| `references/repo-bootstrap.md` | Repository settings | Read-back audit, rulesets, merge choices and PR limits |32| `references/agent-pr-intake.md` | Agent-contributed PRs | Identity, draft policy and supersede proof |33| `references/local-gc.md` | Local worktree cleanup | PR-state evidence, active/dirty guards and dry-run schedule |34| `../jaw-dev/references/stacked-prs.md` | Stacked PRs | Canonical layer and CI rules; opt in to native stack tooling only where supported |35| `references/homebrew.md` | Homebrew distribution | Formula vs Cask, audit/test, livecheck, artifact trust, install/uninstall proof |36| `references/platform-engineering.md` | Platform / DORA / provider routing | DORA capabilities, platform guardrails, provider table rows, SLSA handoff |37| `references/kubernetes.md` | K8s deployment | Gateway API (v1.6+), Kustomize overlays, HPA/VPA, Helm, ArgoCD GitOps |38| `references/ci-cd-deploy.md` | Deploy pipeline, and freeze/GO evidence | GHA reusable workflows, deploy strategies, rollback, GitOps, progressive delivery; §6 suite partitioning, baseline-vs-defect, instrument stability, exact-head evidence, stability counting |39| `references/branch-lifecycle.md` | Branch or worktree cleanup | Deletion-evidence checks, ref snapshot and restore, merged-vs-closed gap, stacked-PR bases, fork identity by repository id, worktree removal order |40| `references/iac.md` | Infrastructure code | OpenTofu/Terraform modules, Pulumi, state encryption, blast radius isolation |41| `references/sre-foundations.md` | Operations/incidents | SLO/SLI/error budget, burn-rate alerting, incident response, blameless postmortem |42| `references/edge-serverless.md` | Edge/serverless work | Edge request shaping, auth at edge, Cloudflare Workers, Vercel Edge, edge AI triage |43| `references/ml-infra.md` | ML infrastructure | GPU cluster mgmt, model registry, scaling, edge inference, MLOps platform patterns |4445Read `package-release.md` for package publishing, registry auth, npm/PyPI46trusted publishing, Bun-to-npm release decisions, and downstream package47channels. Read `cross-platform-release.md` when a release claim depends on48OS-local behavior that CI may not prove. Read `homebrew.md` for Formula/Cask49distribution work. Read `platform-engineering.md` for broader DevOps capability50refresh, DORA, provider routing, and platform guardrails. Read `docker.md` + `ci-cd-deploy.md` first for containerized deploy workflows.51For K8s-specific work, add `kubernetes.md`. For SRE/on-call, add `sre-foundations.md`.5253When release, registry-auth, provider-doc, service-status, image/platform54version, or package-manager behavior depends on current external evidence, read55the active `search` skill and follow its source-fetch and evidence-status rules56instead of relying on stale memory or copied snippets.5758---5960## §1 Container Builds6162### §1.1 Dockerfile Rules (STRICT)6364| Rule | Detail |65|------|--------|66| Multi-stage | Separate build and runtime stages; final image has no build tools |67| Base image | Pin version + SHA256 digest: `node:22-slim@sha256:abc...` |68| Minimal runtime base | Prefer `gcr.io/distroless/*` (Debian-based, keyless-signed) — no shell/package manager; OR Chainguard/Wolfi images when nightly rebuilds, SBOM attestations, patch SLAs, or compliance matter |69| Non-root | `USER nonroot:nonroot` (distroless) or create dedicated user |70| Dependency-first copy | `COPY package.json bun.lock ./` → install → `COPY . .` for layer caching |71| BuildKit secrets | `RUN --mount=type=secret,id=token ...` — never use `ARG` for secrets |72| `.dockerignore` | `.git`, `node_modules`, `.env*`, `*.log`, `dist/`, `coverage/`, `__pycache__/` |7374For canonical Dockerfile templates, read `references/docker.md` §1.7576### §1.2 Image Security & Supply-Chain Baseline (STRICT)7778CRITICAL/HIGH image findings block push. A later checklist exception cannot waive this gate;79any policy change must be approved and declared before the verdict. Other checklist80exceptions follow their owning policy. The 2026 baseline is ONE workflow,81not separate tips: minimal base image (§1.1) → SBOM generation (Syft / Docker Scout) →82vulnerability scan (Trivy or Grype) → sign + attest (Cosign/Sigstore) → digest-based83promote (§2.4). Read `references/docker.md` §4 for scan/SBOM/sign command examples, and84`../jaw-dev-security/references/supply-chain-sbom.md` for deeper SBOM/signing policy.8586### §1.3 Anti-Patterns8788| Banned | Symptom | Fix |89|--------|---------|-----|90| `FROM node:latest` | Irreproducible builds | Pinned version + digest |91| `USER root` in final stage | Attack surface | Non-root user |92| `COPY . .` as first instruction | Cache invalidation on every change | Dependency files first |93| `ARG SECRET=xxx` | Exposed in image history | BuildKit `--mount=type=secret` |94| No scan before push | Vulnerable images in prod | Trivy/Scout CI gate |95| `apt-get install` without cleanup | Bloated image | `--no-install-recommends && rm -rf /var/lib/apt/lists/*` |9697---9899## §2 Deploy Pipeline100101### §2.1 Pipeline Stages (DEFAULT)102103```104[jaw-dev-testing §5] lint → typecheck → test → contract → e2e105[jaw-dev-devops] build-image → scan → push-registry → deploy-staging → smoke → promote → deploy-prod106```107108For stacked PRs, use `../jaw-dev/references/stacked-prs.md` for layer shape109(`DEV-STACK-03`), membership (`DEV-STACK-06`), CI diagnosis (`DEV-STACK-07`),110and the native-tool opt-in (`DEV-STACK-OPT-IN-01`).111112### §2.2 GHA Reusable Workflows (DEFAULT)113114```yaml115# .github/workflows/ci.yml (caller)116jobs:117 build:118 uses: org/templates/.github/workflows/build-test.yml@v2119 with:120 service: payments121 secrets: inherit122```123124| Rule | Detail |125|------|--------|126| `workflow_call` | Central CI template, max 10-level nesting |127| Permissions | Caller cannot escalate; downgrade only |128| Environment | `environment: production` + required reviewers + prevent self-review |129| Promote | Digest-based (`image@sha256:...`), never mutable tags |130131### §2.3 Deploy Strategies (DEFAULT)132133| Strategy | Tool | When | Risk |134|----------|------|------|------|135| Rolling update | K8s Deployment | Stateless, low risk | Low |136| Blue-green | Argo Rollouts `blueGreen:` | Instant rollback, no DB migration | Medium |137| Canary | Argo Rollouts `canary: steps:` | Traffic % control, metric-based promote | Medium |138| Progressive | Flagger | Auto analysis + rollback, A/B testing | Medium-High |139| Feature flag | LaunchDarkly / Unleash | Code-level gradual rollout | Low |140141### §2.4 Rollback Rules (STRICT)142143- Every deployment must be rollback-capable within 5 minutes144- Digest-based promote only — mutable tags are banned145- DB migrations: forward-only + backward-compatible (expand-contract pattern)146- Post-rollback: automatic Slack/PagerDuty notification147148### §2.5 Secret Management (STRICT)149150**Rule (DEVOPS-AUTH-01):** prefer OIDC, workload identity federation, trusted151publishing, or another short-lived credential flow **before** static long-lived152tokens. When a static token is unavoidable, scope it narrowly, store it in the153managed secret system, and rotate on schedule or on incident.154155| Source | Usage |156|--------|-------|157| GHA Secrets / Vault / AWS SM | CI pipeline secrets |158| External Secrets Operator | K8s → Vault/AWS SM sync |159| `.env` files | **Never committed** — generated in CI |160| Rotation | 90-day cycle or immediate on incident |161162### §2.6 GitOps (DEFAULT)163164- **Actions = CI, ArgoCD = CD** — separation of concerns165- Actions updates deploy repo (image digest PR/commit) → ArgoCD reconciles166- Self-heal: ArgoCD auto-reverts drift167- Environment protection: GitHub Environments for prod approval gate168169### §2.7 Release Proof Contract (STRICT)170171**Rule (DEVOPS-RELEASE-PROOF-01):** a release claim must name the artifact digest,172the workflow or builder identity, the deploy target and environment, smoke-test173evidence, and rollback evidence. Five items, each an artifact rather than an174assertion — "deployed successfully" names none of them.175176Keep the proof at this level; detailed package, platform, and supply-chain mechanics177live in the release references.178179### §2.8 Freeze and GO/NO-GO Gates (STRICT)180181`DEVOPS-RELEASE-PROOF-01` governs the proof bundle for an artifact you already182published. This section governs the decision to publish at all — the readiness183report, and the gates it claims to have passed.184185| Rule | Severity | Statement |186|------|----------|-----------|187| `DEVOPS-FREEZE-SHA-01` | STRICT | Pin the readiness report to the code SHA its gates describe. If the head moved after the freeze, prove the delta is docs-only (`git diff --name-only <freeze> <head>`) and keep every gate receipt on the freeze SHA. |188| `DEVOPS-GATE-WEAKEN-01` | STRICT | A red named gate is never excused inside the report that gate failed. Make the original command green, or replace it with a pre-declared equivalent CI actually runs — and declare the swap **before** the verdict, not after the failure. |189| `DEVOPS-REVIEW-THREADS-01` | STRICT | Unresolved review threads on merged PRs are a GO blocker. Count them **after** merge: a thread opened minutes before merge still counts until it is fixed or explicitly dismissed. |190| `DEVOPS-GATE-OWNER-01` | STRICT | A mandatory GO gate needs an implementing work-phase and a recorded terminal outcome — pass, not-reproduced, or explicitly deregistered. A gate nobody implements is not a gate; it is a wish. |191192**Why gate-weakening is the load-bearing one.** In the case that produced these193rules, a freeze audit rejected the first GO report on three counts: unresolved194review threads on merged PRs, a red gate argued into an exception, and missing195frozen-head receipts. The red-gate part is the instructive one — the suite was red,196the red tests were known to be load-sensitive, and the fix was real, so the report197explained the exception. The audit rejected that anyway, and correctly: **an198exception argued *after* a gate fails is indistinguishable from an exception argued199*because* it failed.**200201The honest move was to decompose the gate to match what CI actually runs202(`DEVOPS-SUITE-PARTITION-01`), which turned the general suite green. The one203remaining local failure was waived separately on the `DEVOPS-BASELINE-DEFECT-01`204triple — not by the partitioning. Two different mechanisms for two different205problems; collapsing them is how a partition becomes a way to hide a defect.206207Operational mechanics — suite partitioning, baseline-versus-defect attribution,208instrument stability, and exact-head evidence — live in209`references/ci-cd-deploy.md` §6. Runtime and operator-signal evidence rules live in210`references/sre-foundations.md` §7.211212### §2.9 Branch Lifecycle Hygiene (STRICT)213214This guidance does not itself authorize repository-setting changes, scheduled jobs,215branch or worktree deletion. Apply the task's actual write authority.216217Delivery repositories accumulate dead refs, and the cost is not disk. Stale branches218make the remote branch list unusable for triage, keep superseded heads reachable by219tooling that resolves names, and hide the handful of branches that actually still220matter. Treat branch lifecycle as delivery infrastructure.221222| Rule | Severity | Statement |223|------|----------|-----------|224| `DEVOPS-BRANCH-AUTODELETE-01` | STRICT | Enable host-side head deletion on merge (`delete_branch_on_merge` on GitHub) **and** close the gap it leaves. That setting fires only on merge; a pull request closed without merging keeps its head branch forever, so the closed-PR case needs its own scheduled automation. |225| `DEVOPS-BRANCH-DELETE-EVIDENCE-01` | STRICT | Never bulk-prune. Before deleting any ref, prove per branch that it is not protected, not an open PR head, not the base of an open PR, not a fork head, and not carrying unique commits. A name pattern is not evidence. |226| `DEVOPS-BRANCH-SNAPSHOT-01` | STRICT | Snapshot `git for-each-ref` (SHA + refname) for every local and remote ref to scratch space before the first deletion. A deleted remote branch is restorable with `git push origin <sha>:refs/heads/<name>` only while you still hold the SHA. |227| `DEVOPS-WORKTREE-DIRTY-01` | STRICT | Check every attached worktree for uncommitted work before removing it, and remove worktrees **before** their branches — an attached branch cannot be deleted, and `--force` on a dirty tree discards work no reflog will return. |228| `DEVOPS-BRANCH-NAMESPACE-01` | STRICT | Closed-PR cleanup needs a repository-declared disposable prefix and an exact match between current tip and a related closed PR head SHA; see `branch-lifecycle.md`. |229| `DEVOPS-REPO-BOOTSTRAP-01` | DEFAULT | Read back repository settings before proposing changes; see `repo-bootstrap.md`. |230| `DEVOPS-AGENT-INTAKE-01` | DEFAULT | Use the repository's declared PR review policy; see `agent-pr-intake.md`. |231| `DEVOPS-LOCAL-GC-01` | STRICT | Classify by PR state and SHA; snapshot refs, exclude active/dirty/locked or uncertain trees, and dry-run before cleanup; see `local-gc.md`. |232233**Why the merged/closed distinction is load-bearing.** `delete_branch_on_merge`234reads as complete branch hygiene, so a repository with it enabled looks solved. It235is not: in the case behind this rule the setting was on and the repository still236carried 59 dead remote branches, because closed-unmerged PRs fall outside what it237covers. The failure is silent and compounds — nothing reports it, and the branch238list simply degrades until triage stops using it.239240**Why stacked PRs break naive cleanup.** A stacked child PR targets its parent's241head branch, so deleting a closed parent **closes the open child**. "The PR that242owned this branch is closed" is therefore insufficient grounds for deletion: the243base of any open PR is protected regardless of its own PR state244(`jaw-dev` `DEV-STACK-04`).245246**Fork heads are out of scope, and identity is by repository id.** A fork's head247lives in the contributor's repository; deleting refs there is neither permitted nor248intended. Compare repository **ids**, not names — a fork commonly carries the same249branch names as upstream, so name comparison silently misclassifies it.250251Mechanics, the deletion-plan algorithm, and a worked audit live in252`references/branch-lifecycle.md`. Repository policy choices, contributed PR253review and local worktree cleanup live in `references/repo-bootstrap.md`,254`references/agent-pr-intake.md` and `references/local-gc.md`.255256---257258## §3 Kubernetes Basics259260### §3.1 Minimum Viable K8s (DEFAULT)261262| Resource | Purpose |263|----------|---------|264| Deployment | Pod template + replica management |265| Service | Internal networking |266| HTTPRoute (Gateway API) | External traffic routing — **not Ingress** |267| ResourceQuota | Request/limit enforcement |268| Probes | Liveness + readiness + startup |269| Namespace | Environment isolation (dev/staging/prod) |270271### §3.2 Gateway API (v1.6+, verified 2026-07-02)272273Gateway API is the successor for new routing while Ingress remains GA but feature-frozen274(not planned for removal). v1.6.0 (2026-06) graduates TCPRoute and UDPRoute to GA. Role275separation: platform team owns `GatewayClass` + `Gateway`, app team owns `HTTPRoute`.276Progressive delivery pairs directly with it: Flagger supports Gateway API HTTPRoute canaries.277278```yaml279apiVersion: gateway.networking.k8s.io/v1280kind: HTTPRoute281metadata:282 name: payments-route283spec:284 parentRefs:285 - name: shared-gateway286 hostnames: ["payments.example.com"]287 rules:288 - matches:289 - path: { type: PathPrefix, value: /api }290 backendRefs:291 - name: payments-svc292 port: 8080293```294295### §3.3 Scaling (DEFAULT)296297| Mechanism | Scope | Watch |298|-----------|-------|-------|299| HPA | CPU/memory + custom metrics | Don't combine with VPA on same metric |300| VPA | Request auto-tuning | Use `Off` mode for recommendations only |301| PDB | Disruption budget | `minAvailable: 50%` or `maxUnavailable: 1` |302303### §3.4 Anti-Patterns304305| Banned | Fix |306|--------|-----|307| `Ingress` (new projects) | Gateway API `HTTPRoute` |308| No resource limits | Always set requests + limits |309| `image: app:latest` | SHA digest or pinned SemVer |310| Single replica in prod | Minimum 2 + PDB |311| Secrets in ConfigMap | K8s Secret + External Secrets Operator |312| Annotation-based routing | Gateway API native fields |313314---315316## §4 Infrastructure as Code317318### §4.1 OpenTofu/Terraform Rules (DEFAULT)319320| Rule | Detail |321|------|--------|322| State | Remote backend required (S3+DynamoDB / TF Cloud / OpenTofu) |323| Encryption | OpenTofu native state/plan encryption via KMS |324| Blast radius | Separate state per app/layer/env |325| Modules | Purpose-built (vpc, iam, ecs-service), typed I/O |326| Apply | `plan` → PR review → `apply`; auto-apply staging only |327| Versions | Pin provider + module versions explicitly |328329### §4.2 Tool Selection (HEURISTIC)330331| Tool | Best For | Status (verified 2026-07-02) |332|------|----------|-------------|333| OpenTofu (HCL) | Open-source/licensing-neutral IaC default (MPL-2.0, Linux Foundation; v1.12.x) | ✅ Recommended for OSS neutrality |334| Terraform / HCP Terraform (BSL) | Vendor support or HashiCorp platform integration required | ✅ Active (BSL license — check competitive-use terms) |335| Pulumi (TS/Python) | Teams preferring programming languages | ✅ Active |336| AWS CDK | AWS-only infrastructure | ✅ Active (AWS only) |337| **CDKTF** | — | ❌ Deprecated 2025-12-10; repo archived/read-only, no further fixes |338339### §4.3 Anti-Patterns340341| Banned | Fix |342|--------|-----|343| Local state file | Remote backend required |344| Manual console changes | All changes via code |345| Hardcoded values | Variables + tfvars |346| Monolithic main.tf | Modular decomposition |347| CDKTF (new projects) | OpenTofu or Pulumi |348| Unpinned provider versions | Explicit version constraints |349350---351352## §5 SRE Foundations353354### §5.1 SLO/SLI (DEFAULT)355356| SLI | Measurement | Typical SLO |357|-----|-------------|-------------|358| Availability | Success requests / total | 99.9% (28-day rolling) |359| Latency | p50/p95/p99 response time | p99 < 500ms |360| Error rate | 5xx / total | < 0.1% |361| Freshness | Data update delay | < 5min (pipelines) |362363SLIs measure **user experience**, not infrastructure metrics. "CPU is fine ≠ users are fine."364365Error budget = 1 − SLO (99.9% → 0.1% budget).366367**DORA 2025 (verified 2026-07-02):** AI acts as an *amplifier* — returns depend on the368underlying sociotechnical system. For AI-agent-heavy delivery, invest in golden paths,369guardrails, observability, provenance, and review gates, not just faster merges.370371### §5.2 Error Budget Policy (DEFAULT)372373| Budget State | Action |374|-------------|--------|375| Normal (>50%) | Continue releases, routine monitoring |376| Accelerated burn (20–50%) | Heightened alerts, slow releases, reliability triage |377| **Exhausted (≤0%)** | **Feature freeze** — security/bugfix only; VP exception required |378379Single incident consuming >20% → mandatory postmortem.380Two consecutive window misses → architecture review.381382### §5.3 Incident Response (DEFAULT)3833841. **Detect** → **Triage** (S1/S2/S3) → **Stabilize** → **Fix** → **Postmortem**3852. Roles: IC, Primary Responder, Comms Lead, Scribe3863. **Mitigation first, diagnosis second** during active incidents3874. All S1/S2 → mandatory blameless postmortem within 5 business days3885. Status updates: S1 every 15min, S2 every 30min389390### §5.4 Runbook Template (HEURISTIC)391392```markdown393## [Service] — [Symptom]394### Diagnosis3951. Logs: `kubectl logs -l app=<name> --tail=100`3962. Metrics: Grafana → [dashboard URL]3973. Dependencies: `curl -s http://<dep>/health | jq .`398### Emergency Mitigation3991. Rollback: `argocd app rollback <app>`4002. Traffic block: ...401### Root Fix4021. ...403### Escalation404- Owner: @team-sre405- PagerDuty: [policy]406```407408### §5.5 Anti-Patterns409410| Banned | Fix |411|--------|-----|412| Infrastructure-only SLIs | User-experience-based SLIs |413| SLO without consequences | Error budget policy with freeze gate |414| Too many SLIs (>5 per service) | 2-4 meaningful SLIs |415| Page on every deviation | Burn-rate multi-window alerting |416| Blame individuals | Blameless postmortem, system improvement |417| No error budget policy | Define 3-stage policy (normal/accelerated/exhausted) |418419---420421## §6 Cross-References422423| Topic | Canonical Owner | What jaw-dev-devops defers |424|-------|----------------|----------------------|425| Test strategy & CI test stages | `jaw-dev-testing` §5 | Test pyramid, coverage gates |426| Backend observability code patterns | `jaw-dev-backend` `observability.md` | OTel SDK setup, structured logging |427| Security hardening (app-layer) | `jaw-dev-security` | OWASP, auth, input validation |428| SBOM/signing depth | `jaw-dev-security` `references/supply-chain-sbom.md` | Supply-chain evidence policy beyond image scan gates |429| Architecture module boundaries | `jaw-dev-architecture` | Coupling taxonomy, barrel discipline |430| Scaffolding conventions | `jaw-dev-scaffolding` | File naming, project structure |431| Frontend build/bundle | `jaw-dev-frontend` | Vite/webpack config, SSR |432433**jaw-dev-devops owns**: container builds, deploy pipelines, K8s manifests, IaC modules, SRE/incident response, edge infra, ML infra. DevOps owns operational scan execution and release gates; `jaw-dev-security` owns security policy, severity thresholds, and required evidence.434**jaw-dev-backend owns**: application-layer observability code, API design, health check implementation.435Overlap: observability alerting rules (jaw-dev-devops §5) ↔ observability code instrumentation (jaw-dev-backend `observability.md`). Cross-ref both.436437## Pre-flight Checklist438439Before submitting infrastructure changes:440441- [ ] Dockerfile is multi-stage with distroless/slim final image and non-root user442- [ ] Image scanned (Trivy/Scout) with CRITICAL/HIGH gate — no unresolved findings443- [ ] SBOM generated and attestation signed (Cosign) for production images444- [ ] Deploy pipeline uses digest-based promotion, never mutable tags445- [ ] K8s manifests have resource requests/limits, probes, PDB, and use Gateway API (not Ingress)446- [ ] IaC uses remote state, pinned versions, and modular decomposition447- [ ] Secrets are managed through Vault/AWS SM/GHA Secrets — no `.env` commits, no `ARG` secrets448- [ ] SLO/SLI defined with error budget policy and burn-rate alerting449- [ ] Rollback plan documented and tested — <5min rollback capability confirmed450- [ ] Runbook exists for critical failure scenarios