Deploy Safety

Deploy safety

Deploys go wrong in a small number of ways, and they are all preventable by construction. Every rule below exists because the alternative has shipped.

1. The default inversion is the one that gets you

The most dangerous shape in a repo is two tools with opposite defaults:

deploy.sh             # no flag -> PRODUCTION, staging must be asked for
scripts/backfill.js   # no flag -> staging,    production must be asked for

Both are individually defensible. Together they are a trap, because the muscle memory built by one is exactly wrong for the other. Someone who has run the backfill bare a hundred times will one day run the deploy bare.

Fix it by making the defaults agree. If they genuinely cannot agree, then the mismatch is a documented hazard that belongs at the top of the contributor guide, in those words — not a footnote.

A destructive default is a bug. When in doubt, no flag means the safe environment, and the dangerous one is named out loud on the command line.

2. One place an environment identifier may live

Account ids, table names, bucket names, cluster ARNs, API endpoints: exactly one module resolves them, and nothing else is allowed to contain one.

The failure mode this prevents is real and slow-moving: a constant gets copied into a script, the script gets copied into another script, and a year later forty files name a production resource. Nobody notices, because each of them works. Then someone runs one bare.

Enforce it mechanically:

# CI, before anything else
grep -rE '<production identifier patterns>' --include='*.js' --include='*.ts' . \
  | grep -v 'src/targets.ts' \
  && { echo "production identifier outside the resolver"; exit 1; }

A rule nothing checks is a rule that has already been broken somewhere.

3. Print the target on every run, before doing anything

target: STAGING  (account 1234..., region ap-southeast-2, profile dev)

One line, every script, every mode, including dry runs and --help-adjacent paths. It costs nothing and it is the only thing standing between a reversed assumption and a bad afternoon.

The point is not that someone reads it before pressing enter. The point is that it is in the scrollback afterwards, so when the outcome is surprising, the first question — which environment did that actually touch? — has an answer that is not a guess.

4. Never build the release bundle by hand

Frontend builds inline environment variables at build time. A local .env/.env.local full of localhost URLs, test keys and debug flags will be baked into a bundle that looks completely normal and fails only in the customer's browser.

So:

• The release build runs through the release script, which moves local env files aside, sets the real values, and builds clean.
• After building, grep the output:

grep -rE 'localhost|127\.0\.0\.1|\.local\b' dist/assets/ && echo "LOCAL CONFIG IN BUNDLE"

• Serve index.html and the service worker with no-cache headers, and the hashed assets with long cache. Getting this backwards pins users on an old app that cannot be fixed by shipping.

5. Order the stacks, and mind the chicken and egg

Infrastructure changes have a dependency order, and the tooling will not always work it out:

• Data before app when a new table, index or column feeds a consumer. An app deployed first reads something that is not there yet.
• Index before the query that uses it, in a separate deploy if necessary.
• When the frontend needs a URL that the backend deploy prints, the first deploy is genuinely two passes: deploy to learn the URL, build with it, deploy again. Write that down in the deploy doc; it is not obvious and it is re-derived painfully by every new person.

Where a resource can be updated in place, keep its logical id and scope stable. Renaming the construct is what turns a hitless update into a delete-and-recreate, which for a database is an outage and for a queue is data loss.

6. One deploy at a time

If more than one person or session can deploy, take a lock — a conditional write to a small table, an Actions concurrency group, anything with a TTL and an owner.

The lock needs three things to be usable:

• A status command, so the answer to "is a deploy running?" is one command.
• A TTL, so a crashed deploy does not wedge the pipeline forever.
• An owner and a timestamp in the record, so a stale lock can be judged.

Gate on git sync too: refuse to deploy a working tree that is dirty or behind the integration branch. Deploying uncommitted local work produces a production state that no commit describes.

7. Who may press the button

Production deploys are owned by a person, not by whoever has a shell.

• Trigger production from the release branch, manually, with a typed confirmation.
• Enforce the identity in the cloud role's trust policy, not only in the workflow file. A workflow check is a guard rail; a trust policy is a wall.
• A shared machine's credentials are indistinguishable from the owner's. If an agent session runs on the owner's machine, the deploy button is still the owner's — an agent must not press it without being asked in that specific conversation.

8. After the deploy, verify the thing you deployed

Not "the pipeline went green" — that only proves the pipeline ran.

• Fetch the live asset and confirm the new build hash is being served.
• Hit one endpoint that exercises the change and read the response.
• Check the error rate and the logs for the first few minutes.
• If the change is user-visible, open it.

Checklist

• Safe environment is the default everywhere, or the mismatch is documented as a hazard
• Environment identifiers live in exactly one module, enforced by a CI grep
• Every script prints its target before acting
• Release bundles built by the release script, then grepped for local config
• Stack order correct; logical ids stable
• Deploy lock with status, TTL and owner; git-sync gate
• Production trigger restricted in the cloud trust policy, not just the workflow
• Post-deploy verification hits the live system