Cloud Architecture
Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, +256 784 464178.
Use When
- Use when designing cloud deployments, Dockerising applications, laying out AWS or GCP environments, choosing a deployment pattern, or moving a workload from a single VM to a resilient multi-AZ topology.
Do Not Use When
- The task is unrelated to cloud architecture or would be better handled by a more specific companion skill (
kubernetes-platform for K8s ops, infrastructure-as-code for IaC tooling depth, cicd-pipelines for full pipeline construction).
Required Inputs
- Project context, target users, latency and residency constraints, current stack, and the concrete problem to solve.
- The desired deliverable: design, Dockerfile, compose stack, deploy plan, migration plan, audit, or runbook.
Workflow
- Read this SKILL.md, then load only the referenced deep-dive files relevant to the task.
- Apply the ordered guidance, decision rules, and checklists.
- Produce the deliverable with assumptions, risks, and follow-up work made explicit.
Quality Standards
- Execution-oriented and concise; aligned with
world-class-engineering.
- Self-managed Debian/Ubuntu first, cloud-managed second, in line with the repository's engine stack.
- Deterministic reviewable steps over vague advice or tool-specific magic.
Anti-Patterns
- Jumping to Kubernetes when EC2 + Compose or ECS Fargate would meet the requirement.
- Baking secrets or environment-specific URLs into images.
Outputs
- Workload classification, compute model choice with rationale, VPC + subnet layout, Dockerfile, Compose file, IAM role inventory, deploy pattern + rollback runbook, cost posture, CDN/TLS/WAF/auto-scaling configuration.
- Assumptions, tradeoffs, and unresolved gaps when context is incomplete.
Evidence Produced
| Category |
Artifact |
Format |
Example |
| Correctness |
Cloud topology decision record |
Markdown ADR per skill-composition-standards/references/adr-template.md |
docs/cloud/topology-adr.md |
| Security |
Cloud account hardening checklist |
Markdown doc covering root, IAM, network, logging baseline |
docs/cloud/hardening-checklist.md |
References
references/aws-core-services.md — EC2, S3, RDS, IAM, ALB, ASG, CloudFront CLI recipes.
references/docker-compose-patterns.md — Full local-parity stack template.
references/deployment-patterns.md — Blue-green, rolling, canary runbooks with rollback.
references/github-actions-overview.md — Workflow file structure and reference pipeline.
references/environment-management.md — Staging/production parity and promotion flow.
Load Order
world-class-engineering for the production bar.
system-architecture-design for decomposition and contracts.
- This skill for the cloud runtime shape.
- Pair with
cicd-pipelines for delivery, cicd-devsecops for gate policy, observability-monitoring for telemetry, deployment-release-engineering for rollout, reliability-engineering for failure design, kubernetes-platform for clusters, infrastructure-as-code for IaC depth.
§1 Cloud Foundations & The SaaS-Relevant Subset
The AWS Well-Architected Framework defines six pillars: Operational Excellence, Security, Reliability, Performance Efficiency, Cost Optimization, and Sustainability (aws.amazon.com/architecture/well-architected/). This skill uses these pillars as the spine of its review checklist (§9).
Compute Model Decision Matrix
| Workload pattern |
First choice |
Second choice |
Why |
| Steady web/API, predictable traffic |
Container on Debian/Ubuntu VPS (Compose or systemd) |
EC2 / Compute Engine VM |
Predictable cost, full control |
| Bursty / event-driven (webhooks, schedules) |
Lambda / Cloud Functions |
Containers + autoscaling |
Sub-second cold-start tolerable, pay-per-invocation |
| Long-running background jobs (>15 min) |
Container on VM with queue worker |
ECS / Cloud Run |
Lambda 15-minute hard limit |
| Stateful data layer (MySQL primary) |
Managed RDS / Cloud SQL |
Self-hosted on VPS |
Backup, failover, patching automation |
| Object/file storage |
S3 / Cloud Storage |
Self-hosted MinIO |
Eleven-nines durability, lifecycle policies |
| Multiple services, no platform team |
ECS Fargate with ALB |
Container on VPS |
Managed control plane, ALB integration |
| Polyglot multi-tenant platform |
Kubernetes (kubernetes-platform) |
ECS Fargate |
Workload isolation, per-tenant policies |
Kubernetes is a commitment, not a default.
SaaS-Relevant AWS Subset
- EC2 — compute primitives.
t3/t4g for steady, c6i/c7i CPU-bound, m6i/m7i balanced, r6i/r7i memory-bound, i4i NVMe-heavy.
- S3 — object store for assets, backups, exports. Lifecycle policies move cold data to S3-IA / Glacier.
- RDS — managed MySQL/PostgreSQL with Multi-AZ for production.
- Lambda — event handlers, scheduled jobs, lightweight APIs.
- IAM — identity and policy: least-privilege roles for services, MFA on humans, OIDC for CI.
GCP Equivalents Map
| AWS |
GCP |
| EC2 |
Compute Engine |
| S3 |
Cloud Storage |
| RDS |
Cloud SQL |
| Lambda |
Cloud Functions / Cloud Run |
| IAM |
Cloud IAM |
| ALB |
HTTPS Load Balancer |
| CloudFront |
Cloud CDN |
| ACM |
Certificate Manager |
Cloud Provider Selection (East African Workloads)
| Dimension |
AWS |
GCP |
Azure |
| Closest region |
af-south-1 Cape Town (~30 ms) |
europe-west1 (~160 ms) |
southafricanorth (~40 ms) |
| Data-residency fit |
Strong (af-south-1 + KMS) |
Weak (no ZA region for many services) |
Strong (ZA North + Customer Lockbox) |
| Support in EAT |
24/7 Business; EMEA TAM overlap |
24/7 Standard |
24/7 ProDirect; ZA partners |
| Managed services breadth |
Widest |
Data/ML led |
Microsoft-stack integration |
Default to AWS af-south-1 for Uganda workloads with DPPA 2019 data; use Azure southafricanorth only for .NET-heavy stacks with an existing EA licence; avoid GCP as primary for DPPA-scoped data until a ZA region is GA.
§2 Docker Fundamentals
- Image vs container — image is the read-only template (layers + manifest); container is the running instance with a writable layer on top.
- Layers — each Dockerfile instruction creates a layer. Order from least-frequently-changing (base, system deps) to most-frequently-changing (application code) to maximise cache hits.
- Multi-stage builds — separate
builder (compilers, dev deps) from runtime (slim, no build tools).
- Registries — Docker Hub, GHCR, AWS ECR, GCP Artifact Registry. Tag with both an environment alias and an immutable
:sha-<git-sha> tag; never deploy :latest to production.
- Security basics — run as non-root, scan images with Trivy or Grype, pin base image digest, keep images small (≤200 MB),
.dockerignore excludes .git, node_modules, logs, fixtures.
Production Node.js Dockerfile
# syntax=docker/dockerfile:1.7
FROM node:22.11.0-slim@sha256:<digest> AS builder
WORKDIR /app
COPY package*.json ./
RUN --mount=type=cache,target=/root/.npm npm ci --include=dev
COPY . .
RUN npm run build && npm prune --omit=dev
FROM gcr.io/distroless/nodejs22-debian12:nonroot AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=builder --chown=nonroot:nonroot /app/node_modules ./node_modules
COPY --from=builder --chown=nonroot:nonroot /app/dist ./dist
COPY --from=builder --chown=nonroot:nonroot /app/package.json ./
USER nonroot
EXPOSE 3000
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 CMD ["node", "dist/healthcheck.js"]
CMD ["dist/server.js"]
§3 Docker Compose For App + Dependencies
The Compose Specification consolidates legacy 2.x/3.x file formats; the modern format does not require a version: top-level key. One docker-compose.yml in the repo root mirrors production. Named volumes for stateful services; never bind-mount databases. Declare healthcheck on every dependency and gate startup with depends_on.condition: service_healthy.
services:
app:
build: .
environment:
DATABASE_URL: mysql://app:${DB_PASSWORD}@db:3306/app
REDIS_URL: redis://cache:6379
depends_on:
db: { condition: service_healthy }
cache: { condition: service_started }
healthcheck:
test: ["CMD", "wget", "-qO-", "http://localhost:3000/healthz"]
interval: 10s
retries: 6
restart: unless-stopped
db:
image: mysql:8.4
environment:
MYSQL_DATABASE: app
MYSQL_USER: app
MYSQL_PASSWORD: ${DB_PASSWORD}
MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
volumes: [dbdata:/var/lib/mysql]
healthcheck:
test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
interval: 5s
retries: 10
cache:
image: redis:7-alpine
restart: unless-stopped
proxy:
image: caddy:2
ports: ["80:80", "443:443"]
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile
- caddy_data:/data
depends_on: [app]
volumes:
dbdata: {}
caddy_data: {}
Caddy is recommended for VPS-first deployments because it automates Let's Encrypt issuance and renewal out of the box (caddyserver.com/docs/automatic-https). Commit .env.example, ignore .env, and provide env through the orchestrator in production. See references/docker-compose-patterns.md for the full template.
§4 GitHub Actions CI/CD Overview
Workflows live in .github/workflows/*.yml. Top-level keys: name, on, permissions, env, defaults, concurrency, jobs. Each job needs runs-on and steps; jobs run in parallel by default.
Minimal pattern: build-test job builds the image, runs tests, pushes to GHCR with an immutable sha-<git-sha> tag; deploy-vps job (gated by environment: production for required reviewers) SSHes in and runs docker compose pull && up -d. Secrets discipline: only short-lived deploy credentials in GitHub Actions secrets; long-lived secrets (DB password, API keys) live in Vault and are pulled at runtime.
Full workflow file, cloud-target variants, and concurrency control: references/github-actions-overview.md. Pipeline depth, matrix strategy, reusable workflows: cicd-pipelines. DevSecOps gates: cicd-devsecops.
§5 Staging / Production Environment Management
Four axes of separation:
| Axis |
Staging |
Production |
| Data |
Anonymised production-like fixture; never live PII. |
Live data, encrypted at rest, backed up. |
| Secrets |
Separate Vault path; non-production keys only. |
Vault production path; rotation enforced. |
| Traffic |
Synthetic + internal users. |
Real users; protected by WAF + rate limits. |
| Observability |
Same instrumentation; lower retention; alerts page no one. |
Full retention; on-call paging on SLO-linked alerts. |
Build once, deploy many. The same image SHA that passed staging is the image that runs in production. Configuration differs (env vars, secrets, replica count); the artifact does not. Promotion flow: feature branch → PR + checks → main → staging deploy → smoke + soak → production deploy (same SHA, gated by required reviewers).
Configuration layering, sanitisation script policy, and full promotion checklist: references/environment-management.md.
§6 SSL/TLS, CDN, Auto-Scaling
SSL/TLS Automation
- AWS ALB / CloudFront / API Gateway → ACM certificates: free, auto-renewed, DNS-validated via Route 53. ACM-issued certs cannot be exported.
- VPS-first → Caddy auto-issues and renews from Let's Encrypt with no extra config; nginx + certbot for hosts where Caddy is not viable.
- Kubernetes →
cert-manager with a ClusterIssuer for Let's Encrypt ACME HTTP-01 or DNS-01.
aws acm request-certificate --domain-name app.example.co.ug \
--subject-alternative-names "*.app.example.co.ug" \
--validation-method DNS --key-algorithm RSA_2048
sudo certbot --nginx -d app.example.co.ug --deploy-hook "systemctl reload nginx"
TLS 1.2 minimum, prefer 1.3. Enable HSTS max-age=31536000; includeSubDomains; preload once the production cert path is stable.
CDN
| Goal |
First choice |
Notes |
| AWS-native edge caching |
CloudFront |
Native ACM integration, Lambda@Edge for request rewrite. |
| Multi-cloud or VPS in front |
Cloudflare |
Free tier viable for SaaS MVPs; WAF and bot mitigation included. |
CloudFront or Cloudflare in front of every static asset and cacheable API response. Enable Origin Shield close to origin to cut origin fetches by 60–80%. Attach AWS WAF with the Managed Rules Core Rule Set plus Known Bad Inputs and IP-Reputation; add a rate-based rule at 2000 req/5 min/IP for unauthenticated endpoints. Invalidate surgically — use versioned asset paths (/static/v=<build-sha>/); cache-bust HTML only.
Auto-Scaling
Target tracking first, step scaling second, predictive third. Scale on request count per target and P95 latency — not CPU alone. AWS ASG scales EC2 horizontally on CloudWatch metrics; Lambda concurrency is governed by reserved/provisioned concurrency. For VPS-first with Compose, scale vertically first (bigger VPS), then introduce a load balancer in front of multiple VPS instances. Kubernetes HPA → kubernetes-platform.
aws application-autoscaling put-scaling-policy --service-namespace ecs \
--scalable-dimension ecs:service:DesiredCount --resource-id service/app-cluster/app-svc \
--policy-name tt-reqcount --policy-type TargetTrackingScaling \
--target-tracking-scaling-policy-configuration '{
"TargetValue": 1000,
"PredefinedMetricSpecification": {
"PredefinedMetricType": "ALBRequestCountPerTarget",
"ResourceLabel": "app/alb-arn/tg-arn"
},
"ScaleOutCooldown": 60, "ScaleInCooldown": 300
}'
CPU target 70% for CPU-bound services; never below 40% (wastes capacity). Predictive scaling needs ≥14 days of CloudWatch history and a regular pattern. Warm pools for slow-booting AMIs (>3 min boot).
§7 Zero-Downtime Deployment Patterns
| Pattern |
How it works |
When to use |
Trade-off |
| Rolling |
Replace instances N at a time, health-check each. |
Default for stateless web apps with ≥2 instances. |
Mixed-version window during rollout. |
| Blue/green |
Run new version (green) alongside old (blue); flip traffic via load balancer / DNS. |
Schema-compatible releases needing instant rollback. |
Doubles infra cost during cutover. |
| Canary |
Send a small % of traffic to the new version, expand on green metrics. |
High-risk changes with observability fast enough to detect regression. |
Requires traffic-splitting layer (ALB weighted target groups, Cloudflare LB, service mesh). |
Automatic rollback triggers on health-check failure, 5xx-rate regression > 0.5% over 5 min, or P95 latency regression beyond SLO budget. Schema migrations must be backwards-compatible across two application versions (expand → migrate → contract). Every deploy writes a signed record: who, what, when, artifact digest.
VPS-First Blue/Green With Caddy
- Deploy
app-green on port 3001 alongside app-blue on port 3000 (docker compose --profile green up -d).
- Health-check
app-green for N minutes against /healthz.
- Update Caddy upstream from
:3000 to :3001 and reload (caddy reload --config /etc/caddy/Caddyfile); Caddy reloads without dropping connections.
- Hold blue for a soak window (≥30 minutes) as a hot rollback target.
- Stop
app-blue once error-rate and latency SLOs hold.
Cloud-Managed Blue/Green With ALB
aws elbv2 create-target-group --name app-tg-green --protocol HTTP --port 3000 \
--vpc-id vpc-0abc --health-check-path /healthz --health-check-interval-seconds 15 \
--healthy-threshold-count 2 --unhealthy-threshold-count 3 --matcher HttpCode=200
aws elbv2 modify-listener --listener-arn $LISTENER_ARN \
--default-actions Type=forward,TargetGroupArn=$TG_GREEN
Rolling refresh on ASG:
aws autoscaling start-instance-refresh --auto-scaling-group-name app-prod-asg \
--strategy Rolling --preferences '{
"MinHealthyPercentage": 90, "InstanceWarmup": 180,
"CheckpointPercentages": [25, 50, 100], "CheckpointDelay": 600
}'
Rollback: re-point the listener to app-tg-blue (blue/green) or aws autoscaling cancel-instance-refresh and roll forward with the prior Launch Template version. Full runbooks: references/deployment-patterns.md.
§8 Cost-Aware Architecture Decisions
The Cost Optimization pillar of AWS Well-Architected centres on five design principles: implement cloud financial management, adopt a consumption model, measure overall efficiency, stop spending on undifferentiated heavy lifting, and analyse and attribute expenditure (aws.amazon.com/architecture/well-architected/).
Cost Levers
| Lever |
Action |
Notes |
| Right-sizing |
Match instance class/size to actual CPU+memory profile after ≥2 weeks of metrics. |
Measure before resizing; project must confirm savings. |
| Reserved / Savings Plans |
1- or 3-year commitment for steady-state baseline (70–80% of average compute); spot/preemptible for batch. |
Prefer Compute Savings Plans 1y no-upfront initially; 3y only when headcount and roadmap are certain. |
| S3 lifecycle |
Move logs/backups to S3-IA after 30 d, Glacier after 90 d. |
Storage-class delta; project must measure. |
| Egress |
Keep traffic intra-AZ where possible; CDN absorbs repeat reads. |
NAT GW per AZ avoids cross-AZ data charges. |
| Right-data-tier |
MySQL primary on managed RDS; cold reports → S3 + Athena. |
Avoids overprovisioning RDS. |
| Spot / preemptible |
Async workers, CI runners with graceful shutdown handler for interruption notice. |
Pair with on-demand fallback. |
| Tagging |
Tag every resource with Environment, Team, CostCenter, Project; activate as cost-allocation tags. |
Cost Explorer + per-environment budgets from day one. |
aws ce list-cost-allocation-tags --status Active --region us-east-1
aws budgets create-budget --account-id 111122223333 --budget '{
"BudgetName": "ug-prod-monthly",
"BudgetLimit": { "Amount": "5000", "Unit": "USD" },
"TimeUnit": "MONTHLY", "BudgetType": "COST",
"CostFilters": { "TagKeyValue": ["user:Environment$prod"] }
}'
Verify pricing figures against the current AWS pricing page before publishing — do not quote saving percentages without a fresh source.
§9 Architecture Review Checklist (Six Pillars)
Walk each Well-Architected pillar against the deployment under review:
- Operational Excellence — runbooks documented, deployments automated, postmortems blameless, observability covers all four golden signals (latency, traffic, errors, saturation), telemetry routed to SigNoz.
- Security — IAM least-privilege, secrets in Vault not env files or images, SSL/TLS everywhere including internal hops, audit log retention defined, MFA on every human, root locked away with hardware MFA, OIDC federation for CI.
- Reliability — VPC spans ≥2 AZs, data stores Multi-AZ, backup + restore tested quarterly, RTO/RPO recorded, error budget defined, dependency timeouts and retries explicit.
- Performance Efficiency — instance sizing measured, caching tier present, database indexed for top queries, CDN in front of static assets, P95 latency tracked.
- Cost Optimization — billing alerts on, untagged resources rejected, reserved-vs-on-demand reviewed quarterly, Spot use paired with shutdown handling.
- Sustainability — over-provisioning eliminated, cold storage tiering on, idle dev environments shut down outside work hours, regional choice considers carbon intensity.
Backup, Multi-Region, Security Baseline
These cross-cutting concerns are summarised below; deep CLI is in references/aws-core-services.md.
- Backup & DR — typical SaaS targets RTO ≤ 4 h, RPO ≤ 15 min. RDS automated backups 7–35 days with PITR; weekly manual snapshots retained 90 days; cross-region snapshot copy to
eu-west-1 as a sovereignty-preserving DR site. S3 versioning + Cross-Region Replication for critical buckets. EBS daily snapshots via AWS Backup. Rehearse restore quarterly.
- Multi-Region —
af-south-1 ~30 ms; eu-west-1 ~150 ms; us-east-1 ~220 ms from East Africa. Active-passive (primary af-south-1, warm standby eu-west-1) is the common starting posture; active-active only when conflict-resolution is designed in.
- Account Security Baseline — CloudTrail multi-region with log-file validation and KMS, AWS Config with the Foundational Security Best Practices conformance pack, GuardDuty in every region with S3 and EKS protection, Security Hub aggregating in a delegated admin account, IAM Access Analyzer at organization level reviewed weekly.
Networking & Load Balancers (Quick Reference)
Design VPC across ≥3 AZs for production, 2 for non-production. Allocate /16; carve /20 public and /20 private subnets per AZ. NAT gateway per AZ in production — single-AZ NAT is a SPOF and cross-AZ data charges bite. Security groups (stateful, instance-level) are the primary tool; NACLs (stateless, subnet-level) only for coarse boundaries.
| Feature |
ALB |
NLB |
| Layer |
7 (HTTP/HTTPS/gRPC) |
4 (TCP/UDP/TLS) |
| Routing |
Host, path, header, query |
Port-based |
| TLS termination |
At ALB |
Passthrough or at NLB |
| Use case |
Web APIs, microservices |
High-throughput TCP, static IPs, PrivateLink |
Health checks hit a dedicated /healthz path; verify dependencies shallowly — deep checks cause cascading failures evicting healthy targets. Full networking and AWS-core CLI: references/aws-core-services.md.
Decision rules
| Condition |
Action |
| Single-zone failure breaches SLO |
Use multi-zone placement |
| Managed service fits need and budget |
Prefer managed service |
| Data residency uncertain |
Stop region selection |
Domain Anti-Patterns
- Adding multi-region without recovery need. Fix: derive it from RTO/RPO.
- Exposing databases publicly. Fix: use private networking.
- Giving workloads broad IAM roles. Fix: scope permissions.
- Treating autoscaling as capacity planning. Fix: test limits.
- Ignoring egress and idle cost. Fix: model traffic costs.
Platform Notes
- Codex:
aws CLI and docker CLI are the primary surface. Configure profiles with aws configure sso; use named profiles per environment.
- Codex: treat every command as a patch candidate; keep commands in shell blocks so they stay portable.
1---2name: cloud-architecture3description: Use when designing cloud deployments, Dockerising applications, laying out AWS or GCP environments, choosing a deployment pattern, or moving a workload from a single VM to a resilient multi-AZ topology.4---56# Cloud Architecture78Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, +256 784 464178.910<!-- dual-compat-start -->11## Use When1213- Use when designing cloud deployments, Dockerising applications, laying out AWS or GCP environments, choosing a deployment pattern, or moving a workload from a single VM to a resilient multi-AZ topology.1415## Do Not Use When1617- The task is unrelated to cloud architecture or would be better handled by a more specific companion skill (`kubernetes-platform` for K8s ops, `infrastructure-as-code` for IaC tooling depth, `cicd-pipelines` for full pipeline construction).1819## Required Inputs2021- Project context, target users, latency and residency constraints, current stack, and the concrete problem to solve.22- The desired deliverable: design, Dockerfile, compose stack, deploy plan, migration plan, audit, or runbook.2324## Workflow25261. Read this SKILL.md, then load only the referenced deep-dive files relevant to the task.272. Apply the ordered guidance, decision rules, and checklists.283. Produce the deliverable with assumptions, risks, and follow-up work made explicit.2930## Quality Standards3132- Execution-oriented and concise; aligned with `world-class-engineering`.33- Self-managed Debian/Ubuntu first, cloud-managed second, in line with the repository's engine stack.34- Deterministic reviewable steps over vague advice or tool-specific magic.3536## Anti-Patterns3738- Jumping to Kubernetes when EC2 + Compose or ECS Fargate would meet the requirement.39- Baking secrets or environment-specific URLs into images.4041## Outputs4243- Workload classification, compute model choice with rationale, VPC + subnet layout, Dockerfile, Compose file, IAM role inventory, deploy pattern + rollback runbook, cost posture, CDN/TLS/WAF/auto-scaling configuration.44- Assumptions, tradeoffs, and unresolved gaps when context is incomplete.4546## Evidence Produced4748| Category | Artifact | Format | Example |49|----------|----------|--------|---------|50| Correctness | Cloud topology decision record | Markdown ADR per `skill-composition-standards/references/adr-template.md` | `docs/cloud/topology-adr.md` |51| Security | Cloud account hardening checklist | Markdown doc covering root, IAM, network, logging baseline | `docs/cloud/hardening-checklist.md` |5253## References5455- `references/aws-core-services.md` — EC2, S3, RDS, IAM, ALB, ASG, CloudFront CLI recipes.56- `references/docker-compose-patterns.md` — Full local-parity stack template.57- `references/deployment-patterns.md` — Blue-green, rolling, canary runbooks with rollback.58- `references/github-actions-overview.md` — Workflow file structure and reference pipeline.59- `references/environment-management.md` — Staging/production parity and promotion flow.60<!-- dual-compat-end -->6162## Load Order63641. `world-class-engineering` for the production bar.652. `system-architecture-design` for decomposition and contracts.663. This skill for the cloud runtime shape.674. Pair with `cicd-pipelines` for delivery, `cicd-devsecops` for gate policy, `observability-monitoring` for telemetry, `deployment-release-engineering` for rollout, `reliability-engineering` for failure design, `kubernetes-platform` for clusters, `infrastructure-as-code` for IaC depth.6869## §1 Cloud Foundations & The SaaS-Relevant Subset7071The AWS Well-Architected Framework defines six pillars: Operational Excellence, Security, Reliability, Performance Efficiency, Cost Optimization, and Sustainability (`aws.amazon.com/architecture/well-architected/`). This skill uses these pillars as the spine of its review checklist (§9).7273### Compute Model Decision Matrix7475| Workload pattern | First choice | Second choice | Why |76|------------------|--------------|---------------|-----|77| Steady web/API, predictable traffic | Container on Debian/Ubuntu VPS (Compose or systemd) | EC2 / Compute Engine VM | Predictable cost, full control |78| Bursty / event-driven (webhooks, schedules) | Lambda / Cloud Functions | Containers + autoscaling | Sub-second cold-start tolerable, pay-per-invocation |79| Long-running background jobs (>15 min) | Container on VM with queue worker | ECS / Cloud Run | Lambda 15-minute hard limit |80| Stateful data layer (MySQL primary) | Managed RDS / Cloud SQL | Self-hosted on VPS | Backup, failover, patching automation |81| Object/file storage | S3 / Cloud Storage | Self-hosted MinIO | Eleven-nines durability, lifecycle policies |82| Multiple services, no platform team | ECS Fargate with ALB | Container on VPS | Managed control plane, ALB integration |83| Polyglot multi-tenant platform | Kubernetes (`kubernetes-platform`) | ECS Fargate | Workload isolation, per-tenant policies |8485Kubernetes is a commitment, not a default.8687### SaaS-Relevant AWS Subset8889- EC2 — compute primitives. `t3`/`t4g` for steady, `c6i`/`c7i` CPU-bound, `m6i`/`m7i` balanced, `r6i`/`r7i` memory-bound, `i4i` NVMe-heavy.90- S3 — object store for assets, backups, exports. Lifecycle policies move cold data to S3-IA / Glacier.91- RDS — managed MySQL/PostgreSQL with Multi-AZ for production.92- Lambda — event handlers, scheduled jobs, lightweight APIs.93- IAM — identity and policy: least-privilege roles for services, MFA on humans, OIDC for CI.9495### GCP Equivalents Map9697| AWS | GCP |98|-----|-----|99| EC2 | Compute Engine |100| S3 | Cloud Storage |101| RDS | Cloud SQL |102| Lambda | Cloud Functions / Cloud Run |103| IAM | Cloud IAM |104| ALB | HTTPS Load Balancer |105| CloudFront | Cloud CDN |106| ACM | Certificate Manager |107108### Cloud Provider Selection (East African Workloads)109110| Dimension | AWS | GCP | Azure |111|-----------|-----|-----|-------|112| Closest region | `af-south-1` Cape Town (~30 ms) | `europe-west1` (~160 ms) | `southafricanorth` (~40 ms) |113| Data-residency fit | Strong (af-south-1 + KMS) | Weak (no ZA region for many services) | Strong (ZA North + Customer Lockbox) |114| Support in EAT | 24/7 Business; EMEA TAM overlap | 24/7 Standard | 24/7 ProDirect; ZA partners |115| Managed services breadth | Widest | Data/ML led | Microsoft-stack integration |116117Default to AWS `af-south-1` for Uganda workloads with DPPA 2019 data; use Azure `southafricanorth` only for .NET-heavy stacks with an existing EA licence; avoid GCP as primary for DPPA-scoped data until a ZA region is GA.118119## §2 Docker Fundamentals120121- Image vs container — image is the read-only template (layers + manifest); container is the running instance with a writable layer on top.122- Layers — each Dockerfile instruction creates a layer. Order from least-frequently-changing (base, system deps) to most-frequently-changing (application code) to maximise cache hits.123- Multi-stage builds — separate `builder` (compilers, dev deps) from `runtime` (slim, no build tools).124- Registries — Docker Hub, GHCR, AWS ECR, GCP Artifact Registry. Tag with both an environment alias and an immutable `:sha-<git-sha>` tag; never deploy `:latest` to production.125- Security basics — run as non-root, scan images with Trivy or Grype, pin base image digest, keep images small (≤200 MB), `.dockerignore` excludes `.git`, `node_modules`, logs, fixtures.126127### Production Node.js Dockerfile128129```dockerfile130# syntax=docker/dockerfile:1.7131FROM node:22.11.0-slim@sha256:<digest> AS builder132WORKDIR /app133COPY package*.json ./134RUN --mount=type=cache,target=/root/.npm npm ci --include=dev135COPY . .136RUN npm run build && npm prune --omit=dev137138FROM gcr.io/distroless/nodejs22-debian12:nonroot AS runtime139WORKDIR /app140ENV NODE_ENV=production141COPY --from=builder --chown=nonroot:nonroot /app/node_modules ./node_modules142COPY --from=builder --chown=nonroot:nonroot /app/dist ./dist143COPY --from=builder --chown=nonroot:nonroot /app/package.json ./144USER nonroot145EXPOSE 3000146HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 CMD ["node", "dist/healthcheck.js"]147CMD ["dist/server.js"]148```149150## §3 Docker Compose For App + Dependencies151152The Compose Specification consolidates legacy 2.x/3.x file formats; the modern format does not require a `version:` top-level key. One `docker-compose.yml` in the repo root mirrors production. Named volumes for stateful services; never bind-mount databases. Declare `healthcheck` on every dependency and gate startup with `depends_on.condition: service_healthy`.153154```yaml155services:156 app:157 build: .158 environment:159 DATABASE_URL: mysql://app:${DB_PASSWORD}@db:3306/app160 REDIS_URL: redis://cache:6379161 depends_on:162 db: { condition: service_healthy }163 cache: { condition: service_started }164 healthcheck:165 test: ["CMD", "wget", "-qO-", "http://localhost:3000/healthz"]166 interval: 10s167 retries: 6168 restart: unless-stopped169170 db:171 image: mysql:8.4172 environment:173 MYSQL_DATABASE: app174 MYSQL_USER: app175 MYSQL_PASSWORD: ${DB_PASSWORD}176 MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}177 volumes: [dbdata:/var/lib/mysql]178 healthcheck:179 test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]180 interval: 5s181 retries: 10182183 cache:184 image: redis:7-alpine185 restart: unless-stopped186187 proxy:188 image: caddy:2189 ports: ["80:80", "443:443"]190 volumes:191 - ./Caddyfile:/etc/caddy/Caddyfile192 - caddy_data:/data193 depends_on: [app]194195volumes:196 dbdata: {}197 caddy_data: {}198```199200Caddy is recommended for VPS-first deployments because it automates Let's Encrypt issuance and renewal out of the box (`caddyserver.com/docs/automatic-https`). Commit `.env.example`, ignore `.env`, and provide env through the orchestrator in production. See `references/docker-compose-patterns.md` for the full template.201202## §4 GitHub Actions CI/CD Overview203204Workflows live in `.github/workflows/*.yml`. Top-level keys: `name`, `on`, `permissions`, `env`, `defaults`, `concurrency`, `jobs`. Each job needs `runs-on` and `steps`; jobs run in parallel by default.205206Minimal pattern: `build-test` job builds the image, runs tests, pushes to GHCR with an immutable `sha-<git-sha>` tag; `deploy-vps` job (gated by `environment: production` for required reviewers) SSHes in and runs `docker compose pull && up -d`. Secrets discipline: only short-lived deploy credentials in GitHub Actions secrets; long-lived secrets (DB password, API keys) live in Vault and are pulled at runtime.207208Full workflow file, cloud-target variants, and concurrency control: `references/github-actions-overview.md`. Pipeline depth, matrix strategy, reusable workflows: `cicd-pipelines`. DevSecOps gates: `cicd-devsecops`.209210## §5 Staging / Production Environment Management211212Four axes of separation:213214| Axis | Staging | Production |215|------|---------|------------|216| Data | Anonymised production-like fixture; never live PII. | Live data, encrypted at rest, backed up. |217| Secrets | Separate Vault path; non-production keys only. | Vault production path; rotation enforced. |218| Traffic | Synthetic + internal users. | Real users; protected by WAF + rate limits. |219| Observability | Same instrumentation; lower retention; alerts page no one. | Full retention; on-call paging on SLO-linked alerts. |220221**Build once, deploy many.** The same image SHA that passed staging is the image that runs in production. Configuration differs (env vars, secrets, replica count); the artifact does not. Promotion flow: feature branch → PR + checks → main → staging deploy → smoke + soak → production deploy (same SHA, gated by required reviewers).222223Configuration layering, sanitisation script policy, and full promotion checklist: `references/environment-management.md`.224225## §6 SSL/TLS, CDN, Auto-Scaling226227### SSL/TLS Automation228229- AWS ALB / CloudFront / API Gateway → ACM certificates: free, auto-renewed, DNS-validated via Route 53. ACM-issued certs cannot be exported.230- VPS-first → Caddy auto-issues and renews from Let's Encrypt with no extra config; nginx + certbot for hosts where Caddy is not viable.231- Kubernetes → `cert-manager` with a `ClusterIssuer` for Let's Encrypt ACME HTTP-01 or DNS-01.232233```bash234aws acm request-certificate --domain-name app.example.co.ug \235 --subject-alternative-names "*.app.example.co.ug" \236 --validation-method DNS --key-algorithm RSA_2048237sudo certbot --nginx -d app.example.co.ug --deploy-hook "systemctl reload nginx"238```239240TLS 1.2 minimum, prefer 1.3. Enable HSTS `max-age=31536000; includeSubDomains; preload` once the production cert path is stable.241242### CDN243244| Goal | First choice | Notes |245|------|--------------|-------|246| AWS-native edge caching | CloudFront | Native ACM integration, Lambda@Edge for request rewrite. |247| Multi-cloud or VPS in front | Cloudflare | Free tier viable for SaaS MVPs; WAF and bot mitigation included. |248249CloudFront or Cloudflare in front of every static asset and cacheable API response. Enable Origin Shield close to origin to cut origin fetches by 60–80%. Attach AWS WAF with the Managed Rules Core Rule Set plus Known Bad Inputs and IP-Reputation; add a rate-based rule at 2000 req/5 min/IP for unauthenticated endpoints. Invalidate surgically — use versioned asset paths (`/static/v=<build-sha>/`); cache-bust HTML only.250251### Auto-Scaling252253Target tracking first, step scaling second, predictive third. Scale on request count per target and P95 latency — not CPU alone. AWS ASG scales EC2 horizontally on CloudWatch metrics; Lambda concurrency is governed by reserved/provisioned concurrency. For VPS-first with Compose, scale vertically first (bigger VPS), then introduce a load balancer in front of multiple VPS instances. Kubernetes HPA → `kubernetes-platform`.254255```bash256aws application-autoscaling put-scaling-policy --service-namespace ecs \257 --scalable-dimension ecs:service:DesiredCount --resource-id service/app-cluster/app-svc \258 --policy-name tt-reqcount --policy-type TargetTrackingScaling \259 --target-tracking-scaling-policy-configuration '{260 "TargetValue": 1000,261 "PredefinedMetricSpecification": {262 "PredefinedMetricType": "ALBRequestCountPerTarget",263 "ResourceLabel": "app/alb-arn/tg-arn"264 },265 "ScaleOutCooldown": 60, "ScaleInCooldown": 300266 }'267```268269CPU target 70% for CPU-bound services; never below 40% (wastes capacity). Predictive scaling needs ≥14 days of CloudWatch history and a regular pattern. Warm pools for slow-booting AMIs (>3 min boot).270271## §7 Zero-Downtime Deployment Patterns272273| Pattern | How it works | When to use | Trade-off |274|---------|--------------|-------------|-----------|275| Rolling | Replace instances N at a time, health-check each. | Default for stateless web apps with ≥2 instances. | Mixed-version window during rollout. |276| Blue/green | Run new version (green) alongside old (blue); flip traffic via load balancer / DNS. | Schema-compatible releases needing instant rollback. | Doubles infra cost during cutover. |277| Canary | Send a small % of traffic to the new version, expand on green metrics. | High-risk changes with observability fast enough to detect regression. | Requires traffic-splitting layer (ALB weighted target groups, Cloudflare LB, service mesh). |278279Automatic rollback triggers on health-check failure, 5xx-rate regression > 0.5% over 5 min, or P95 latency regression beyond SLO budget. Schema migrations must be backwards-compatible across two application versions (expand → migrate → contract). Every deploy writes a signed record: who, what, when, artifact digest.280281### VPS-First Blue/Green With Caddy2822831. Deploy `app-green` on port 3001 alongside `app-blue` on port 3000 (`docker compose --profile green up -d`).2842. Health-check `app-green` for N minutes against `/healthz`.2853. Update Caddy upstream from `:3000` to `:3001` and reload (`caddy reload --config /etc/caddy/Caddyfile`); Caddy reloads without dropping connections.2864. Hold blue for a soak window (≥30 minutes) as a hot rollback target.2875. Stop `app-blue` once error-rate and latency SLOs hold.288289### Cloud-Managed Blue/Green With ALB290291```bash292aws elbv2 create-target-group --name app-tg-green --protocol HTTP --port 3000 \293 --vpc-id vpc-0abc --health-check-path /healthz --health-check-interval-seconds 15 \294 --healthy-threshold-count 2 --unhealthy-threshold-count 3 --matcher HttpCode=200295aws elbv2 modify-listener --listener-arn $LISTENER_ARN \296 --default-actions Type=forward,TargetGroupArn=$TG_GREEN297```298299Rolling refresh on ASG:300301```bash302aws autoscaling start-instance-refresh --auto-scaling-group-name app-prod-asg \303 --strategy Rolling --preferences '{304 "MinHealthyPercentage": 90, "InstanceWarmup": 180,305 "CheckpointPercentages": [25, 50, 100], "CheckpointDelay": 600306 }'307```308309Rollback: re-point the listener to `app-tg-blue` (blue/green) or `aws autoscaling cancel-instance-refresh` and roll forward with the prior Launch Template version. Full runbooks: `references/deployment-patterns.md`.310311## §8 Cost-Aware Architecture Decisions312313The Cost Optimization pillar of AWS Well-Architected centres on five design principles: implement cloud financial management, adopt a consumption model, measure overall efficiency, stop spending on undifferentiated heavy lifting, and analyse and attribute expenditure (`aws.amazon.com/architecture/well-architected/`).314315### Cost Levers316317| Lever | Action | Notes |318|-------|--------|-------|319| Right-sizing | Match instance class/size to actual CPU+memory profile after ≥2 weeks of metrics. | Measure before resizing; project must confirm savings. |320| Reserved / Savings Plans | 1- or 3-year commitment for steady-state baseline (70–80% of average compute); spot/preemptible for batch. | Prefer Compute Savings Plans 1y no-upfront initially; 3y only when headcount and roadmap are certain. |321| S3 lifecycle | Move logs/backups to S3-IA after 30 d, Glacier after 90 d. | Storage-class delta; project must measure. |322| Egress | Keep traffic intra-AZ where possible; CDN absorbs repeat reads. | NAT GW per AZ avoids cross-AZ data charges. |323| Right-data-tier | MySQL primary on managed RDS; cold reports → S3 + Athena. | Avoids overprovisioning RDS. |324| Spot / preemptible | Async workers, CI runners with graceful shutdown handler for interruption notice. | Pair with on-demand fallback. |325| Tagging | Tag every resource with `Environment`, `Team`, `CostCenter`, `Project`; activate as cost-allocation tags. | Cost Explorer + per-environment budgets from day one. |326327```bash328aws ce list-cost-allocation-tags --status Active --region us-east-1329aws budgets create-budget --account-id 111122223333 --budget '{330 "BudgetName": "ug-prod-monthly",331 "BudgetLimit": { "Amount": "5000", "Unit": "USD" },332 "TimeUnit": "MONTHLY", "BudgetType": "COST",333 "CostFilters": { "TagKeyValue": ["user:Environment$prod"] }334}'335```336337Verify pricing figures against the current AWS pricing page before publishing — do not quote saving percentages without a fresh source.338339## §9 Architecture Review Checklist (Six Pillars)340341Walk each Well-Architected pillar against the deployment under review:342343- **Operational Excellence** — runbooks documented, deployments automated, postmortems blameless, observability covers all four golden signals (latency, traffic, errors, saturation), telemetry routed to SigNoz.344- **Security** — IAM least-privilege, secrets in Vault not env files or images, SSL/TLS everywhere including internal hops, audit log retention defined, MFA on every human, root locked away with hardware MFA, OIDC federation for CI.345- **Reliability** — VPC spans ≥2 AZs, data stores Multi-AZ, backup + restore tested quarterly, RTO/RPO recorded, error budget defined, dependency timeouts and retries explicit.346- **Performance Efficiency** — instance sizing measured, caching tier present, database indexed for top queries, CDN in front of static assets, P95 latency tracked.347- **Cost Optimization** — billing alerts on, untagged resources rejected, reserved-vs-on-demand reviewed quarterly, Spot use paired with shutdown handling.348- **Sustainability** — over-provisioning eliminated, cold storage tiering on, idle dev environments shut down outside work hours, regional choice considers carbon intensity.349350## Backup, Multi-Region, Security Baseline351352These cross-cutting concerns are summarised below; deep CLI is in `references/aws-core-services.md`.353354- **Backup & DR** — typical SaaS targets RTO ≤ 4 h, RPO ≤ 15 min. RDS automated backups 7–35 days with PITR; weekly manual snapshots retained 90 days; cross-region snapshot copy to `eu-west-1` as a sovereignty-preserving DR site. S3 versioning + Cross-Region Replication for critical buckets. EBS daily snapshots via AWS Backup. Rehearse restore quarterly.355- **Multi-Region** — `af-south-1` ~30 ms; `eu-west-1` ~150 ms; `us-east-1` ~220 ms from East Africa. Active-passive (primary `af-south-1`, warm standby `eu-west-1`) is the common starting posture; active-active only when conflict-resolution is designed in.356- **Account Security Baseline** — CloudTrail multi-region with log-file validation and KMS, AWS Config with the Foundational Security Best Practices conformance pack, GuardDuty in every region with S3 and EKS protection, Security Hub aggregating in a delegated admin account, IAM Access Analyzer at organization level reviewed weekly.357358## Networking & Load Balancers (Quick Reference)359360Design VPC across ≥3 AZs for production, 2 for non-production. Allocate /16; carve /20 public and /20 private subnets per AZ. NAT gateway per AZ in production — single-AZ NAT is a SPOF and cross-AZ data charges bite. Security groups (stateful, instance-level) are the primary tool; NACLs (stateless, subnet-level) only for coarse boundaries.361362| Feature | ALB | NLB |363|---------|-----|-----|364| Layer | 7 (HTTP/HTTPS/gRPC) | 4 (TCP/UDP/TLS) |365| Routing | Host, path, header, query | Port-based |366| TLS termination | At ALB | Passthrough or at NLB |367| Use case | Web APIs, microservices | High-throughput TCP, static IPs, PrivateLink |368369Health checks hit a dedicated `/healthz` path; verify dependencies shallowly — deep checks cause cascading failures evicting healthy targets. Full networking and AWS-core CLI: `references/aws-core-services.md`.370371## Decision rules372| Condition | Action |373|---|---|374| Single-zone failure breaches SLO | Use multi-zone placement |375| Managed service fits need and budget | Prefer managed service |376| Data residency uncertain | Stop region selection |377378## Domain Anti-Patterns379- Adding multi-region without recovery need. Fix: derive it from RTO/RPO.380- Exposing databases publicly. Fix: use private networking.381- Giving workloads broad IAM roles. Fix: scope permissions.382- Treating autoscaling as capacity planning. Fix: test limits.383- Ignoring egress and idle cost. Fix: model traffic costs.384385## Platform Notes386387- Codex: `aws` CLI and `docker` CLI are the primary surface. Configure profiles with `aws configure sso`; use named profiles per environment.388- Codex: treat every command as a patch candidate; keep commands in shell blocks so they stay portable.