Secrets Management

secrets-management

A leaked secret isn't a bug you fix by deleting a line — it's a credential an attacker may already have, in a place (git history, an image layer, a log aggregator) that keeps it long after you "removed" it. This skill keeps secrets out of those places to begin with, and handles the leak-response when one gets in anyway.

The injection hierarchy (best → worst)

1. No long-lived secret at all — workload identity / OIDC federation. The workload proves who it is and gets a short-lived token minted on demand. Nothing static to leak. Prefer this everywhere it's available (cloud SDKs, CI→cloud, service→service).
2. A secrets manager (Vault, AWS/GCP Secrets Manager, SOPS-encrypted files) the app reads at startup or on demand, authenticated by workload identity. Central rotation, audit log, access control.
3. Injected environment variables / mounted files from the platform's secret store (K8s Secret mounted as a file > as an env var — env vars leak into child processes, crash dumps, and /proc).
4. A committed .env file — only for local dev, never for shared/prod, and .env is in .gitignore from the first commit.

Never: a secret hardcoded in source, in a client-side bundle (anything shipped to a browser/app is public — there are no frontend secrets), in a Dockerfile ENV/ARG, in a committed config file, or in CI plaintext where a fork PR could read it.

Keeping them out

• .gitignore / .dockerignore before the first commit. .env, *.pem, *.key, credential JSONs, *.tfvars with secrets.
• A pre-commit secret scanner (gitleaks/trufflehog) so a key never reaches a commit. Add the same scan in CI as a backstop.
• Redact in logs. Build a logging serializer that masks known-sensitive keys (authorization, password, token, secret, api_key) — don't rely on remembering at each call site. Assume logs ship to a third-party backend.
• Not in images/state: no secret in a Docker layer (use BuildKit secret mounts), no secret in Terraform state committed to git (state holds secrets in plaintext — encrypt the backend, keep it out of git).
• Not in URLs. A secret in a query string lands in access logs, browser history, and referrer headers. Use headers/body.

When a secret leaks — the response

Order matters. Do these in sequence:

1. Rotate/revoke first. The secret is compromised the moment it's exposed. Invalidate it and issue a new one before anything else. Scrubbing history without rotating is theater — the value is already out.
2. Assess exposure. Where did it go — public repo, a log aggregator, an image on a registry, a Slack message? Each is a separate copy to deal with.
3. Purge from history if needed (git filter-repo / BFG) and force-push — but only after rotation, and know that anyone who cloned/forked, and GitHub's cached views, may still have it. History rewrite is cleanup, not containment.
4. Audit for use. Check access logs for the credential between exposure and revocation.
5. Add the guard that would have caught it (pre-commit scanner, review rule) so it doesn't recur.

"Removed in a later commit/layer" ≠ gone. A secret committed and then deleted is still in git history. A secret added and rm-ed in a later Docker layer is still in the earlier layer. The only fix is rotation + purge, not deletion.

Rotation

• Secrets have a lifecycle. Rotate on a schedule and on personnel changes, not just after incidents. Short-lived (hours) beats long-lived (years) — the shorter the life, the smaller the leak window.
• Design for zero-downtime rotation: support two valid secrets at once (old + new) during the cutover so rotating doesn't cause an outage. A secret you can't rotate without downtime won't get rotated.
• Prefer systems that rotate automatically (managed DB credentials, dynamic secrets from Vault) over manual rotation you'll forget.

Procedure

1. Classify how the secret should be delivered using the hierarchy — push toward workload identity / short-lived.
2. Wire the app to read from the secret manager / injected mount, authenticated by workload identity; remove any hardcoded/committed value.
3. Ensure .gitignore/.dockerignore cover it; add a pre-commit + CI secret scanner; add log redaction.
4. Confirm it's absent from source, client bundles, image layers, Terraform-state-in-git, URLs, and logs.
5. Set up rotation (scheduled or automatic) with dual-secret cutover.
6. If responding to a leak: rotate → assess → purge → audit → add the guard, in that order.

Definition of done

• The secret is delivered by the highest-feasible tier (ideally short-lived/workload identity), never hardcoded or committed.
• Ignore files + pre-commit + CI scanning prevent commits; logs redact sensitive keys.
• Not present in source, client bundle, image layers, state-in-git, URLs, or logs.
• A rotation path exists and supports overlap (no-downtime cutover).
• Leak response, if any, rotated before scrubbing and audited access.