Devops Platform

DevOps and Platform Engineering

Two questions decide whether a platform is professional:

How long from commit to production? How long from "this is broken" to it not being broken?

Everything below serves those two numbers. A pipeline that produces neither speed nor reversibility is ceremony.

Reproducibility — the precondition

If the environment cannot be rebuilt from the repository, everything else is built on sand.

• Runtime version pinned in a file the toolchain reads, and in the manifest.
• Lockfile committed; CI installs frozen (npm ci, --frozen-lockfile, poetry install --sync, bundle --deployment).
• Container base images pinned by digest, not by a moving tag.
• Infrastructure in code, in the repository, applied by the pipeline.
• No snowflake servers. Anything configured by hand once is a single point of failure with no recovery procedure. The test: could you delete the production environment and rebuild it from the repo?
• Local development one command away — Docker Compose, devcontainer, or a documented single script.

The pipeline

On every pull request, blocking merge:

install (frozen)  →  lint  →  typecheck  →  unit  →  integration  →  build  →  dependency audit  →  secret scan

On merge to the default branch:

build once, tag by commit SHA  →  deploy to staging  →  smoke tests  →  ready for production

Rules that matter more than the tool:

• Build the artefact once. Promote the same artefact through environments. Rebuilding per environment means you deployed something you never tested.
• CI must block merge. Establish this before there is pressure to bypass it; it is never established afterwards.
• Keep it under ~10 minutes. Beyond that, people stop waiting and start merging around it. Cache dependencies, parallelise, split slow suites.
• A flaky pipeline is a broken pipeline. A suite people re-run until green provides no information.
• Pin actions to a commit SHA, not a tag. Tags are mutable.
• Fork PRs must not access secrets.
• Branch protection: no direct pushes, review required, checks required.

Environments

Local, staging and production should be structurally identical, differing in scale. Staging with a different database engine, no data and half the services stubbed does not tell you anything about production.

Staging must never hold production personal data or production credentials. If staging can read the production database, staging is production for breach purposes, with weaker controls and broader access. See database-engineering/references/data-lifecycle.md.

Ephemeral preview environments per pull request are the highest-value platform investment most teams have not made: they turn "looks right in the diff" into "I clicked it".

Deployment and rollback

Rollback is the feature. A deploy you cannot reverse in minutes is a deploy you are afraid to make, which is how teams end up shipping monthly.

Non-negotiables

• The rollback procedure is documented and has been executed at least once for practice. An untested rollback has unknown duration.
• Migrations are ordered relative to the deploy by direction: additive changes migrate first, destructive changes deploy first. During a rolling deploy both code versions run simultaneously, so the schema must satisfy both.
• Health and readiness endpoints are distinct: liveness means "restart me", readiness means "do not send traffic yet".
• Graceful shutdown on SIGTERM, draining in-flight requests. Without it, every deploy drops requests.
• Deploys are logged and annotated on dashboards — the first question in any incident is "what changed?"

Containers

# Multi-stage: build tooling never reaches the runtime image
FROM node:22.11.0-slim@sha256:<digest> AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci --ignore-scripts
COPY . .
RUN npm run build && npm prune --omit=dev

FROM node:22.11.0-slim@sha256:<digest>
ENV NODE_ENV=production
WORKDIR /app
RUN groupadd -r app && useradd -r -g app app
COPY --from=build --chown=app:app /app/node_modules ./node_modules
COPY --from=build --chown=app:app /app/dist ./dist
USER app                                    # never root
EXPOSE 3000
HEALTHCHECK --interval=30s --timeout=3s CMD node dist/healthcheck.js
CMD ["node", "dist/server.js"]

• Pin the base image by digest. Multi-stage so compilers and dev dependencies do not ship.
• Run as non-root. Read-only root filesystem where possible.
• No secrets in layers, ever — including in intermediate layers a later RM "removed". Layers are permanently inspectable. Use build secrets or runtime injection.
• .dockerignore covering .git, .env, node_modules, tests.
• Scan images for vulnerabilities in CI.
• Set memory and CPU limits, and make sure the runtime respects them (older runtimes ignore cgroup limits and get OOM-killed).

Secrets

• In the platform's secret manager (cloud KMS/Secrets Manager, Vault, or the hosting platform's own). Never in the repository, never in the image, never in CI logs.
• Injected at runtime, distinct per environment.
• Rotatable, with a documented and practised procedure. A secret nobody knows how to rotate is a secret you cannot revoke after a leak.
• Access audited. Least privilege per service.
• Prefer short-lived federated credentials (OIDC from CI to cloud) over long-lived static keys.
• Secret scanning in CI and as a pre-commit hook.

Disaster recovery

Backups and restore drills live in database-engineering/references/resilience.md — that is the core of DR and it is the part most often unverified.

Beyond the database:

• Object storage versioned and backed up; deletion protection on.
• Infrastructure rebuildable from code — tested by actually creating a fresh environment from scratch.
• Secrets recoverable (who else can access the manager if one person is unavailable?).
• DNS and domain ownership documented, with registrar access not held by one person.
• TLS certificates auto-renewing, with expiry alerting as a backstop.
• A written incident runbook, and a named escalation path.
• Dependencies on third parties assessed: what breaks if the identity provider, payment processor or CDN is down?

Cost

Cost is an engineering property, not a finance problem discovered at month end.

• Tag every resource by service and environment; report spend per tag.
• Alert on anomalies, not just on budget — a 3x jump on Tuesday matters more than a monthly total.
• The usual leaks, in order: unbounded log retention; over-provisioned instances running at 5%; forgotten non-production environments; cross-AZ and egress traffic; unattached volumes and old snapshots; oversized managed databases.
• Autoscaling with a maximum — unbounded scaling turns a traffic spike or a retry storm into a five-figure bill.
• Set a budget alert on day one, before anything is deployed.

Review checklist

• Environment rebuildable from the repository alone
• Lockfile committed; CI installs frozen; images pinned by digest
• CI runs lint, types, tests, build, dependency audit, secret scan — blocking
• CI under ~10 minutes and not flaky
• One artefact built once and promoted
• Staging structurally matches production and holds no production data
• Deploys automated; rollback documented and rehearsed
• Migration ordering relative to deploys is correct for both directions
• Health and readiness distinct; graceful shutdown implemented
• Containers non-root, multi-stage, no secrets in layers, scanned
• Secrets in a manager, per-environment, rotatable, audited
• Backups tested by restore; infrastructure rebuild tested
• Cost tagged, monitored, anomaly-alerted; autoscaling capped

References

• references/pipeline-design.md — CI/CD in depth, per platform
• references/infrastructure.md — IaC, environments, networking, scaling, cost