Secrets Management

secrets-management

Decision framework for storing, rotating, and scanning secrets across cloud, CI, and Kubernetes. Provider deep-dives live upstream (links in § Provenance) — this skill is the predicate, not the per-vendor cookbook. Sunset-policy compliant.

When to use

• Designing where a new secret lives (env var, Vault, AWS Secrets Manager, Azure KV, GCP SM, GitHub/GitLab CI, k8s).
• Reviewing a diff that introduces a credential, API key, signing key, or DB password.
• Setting up secret rotation for an existing application.
• Wiring secret-scanning gates into pre-commit, CI, or org-policy.

Do NOT use when:

• The secret is project-AWS-only and aws-infrastructure already covers the placement — route there.
• The work is a security audit of running code — route to security-audit or threat-modeling.
• The decision is which cipher to use for at-rest encryption — read the provider's KMS docs directly.

Decision framework

Step 1 — Pick the store

Ephemeral, dev-only, never leaves the laptop  → .env (gitignored). Stop.
Single-cloud, single-app                       → that cloud's native store
                                                  (AWS Secrets Manager / Azure Key Vault /
                                                  GCP Secret Manager).
Multi-cloud OR on-prem hybrid                  → HashiCorp Vault (or cloud-agnostic equivalent).
CI-only (deploy keys, signing tokens)          → GitHub/GitLab repo+environment secrets;
                                                  scope to environment (production/staging).
Kubernetes workload secrets                    → External Secrets Operator pulling from
                                                  the canonical store above; never hand-rolled
                                                  k8s `Secret` objects committed to git.
Cross-tenant / cross-org shared secret         → don't. Re-architect; shared secrets are an
                                                  outage and a breach class on their own.
AI-agent / MCP-server credential               → env-var indirection in the agent/MCP config
                                                  (`.mcp.json`, agent-config, CI YAML) — the
                                                  value lives in the store above; the config
                                                  holds only `${ENV_VAR}`. Never a raw key
                                                  inline, and the config path is gitignored if
                                                  it can hold one.

Step 2 — Pick the access pattern

Application reads at boot                      → fetch once, hold in memory; re-fetch on rotation event.
Application reads per-request                  → cache with TTL ≤ rotation period / 2.
Short-lived workload (Lambda, Job)             → fetch per-invocation; rely on platform IAM.
Long-lived workload                            → leased / dynamic credentials (Vault DB engine,
                                                  AWS IAM role) — never static creds.
Human access                                   → no shared logins; per-user identity + audit.

Step 3 — Define the rotation contract

Every secret MUST have:

• Owner — team/person responsible for rotation; tracked in code or runbook.
• Period — calendar trigger (e.g. 90 days for static creds, hours/minutes for dynamic).
• Mechanism — automated (Lambda + Secrets Manager rotation, Vault dynamic secret, IAM role) or documented manual procedure.
• Verification — post-rotation health check; alert if old credential still observed in use after rotation grace window.

Static secrets without a rotation mechanism are a deferred incident — refuse to merge.

Step 4 — Wire the scanning gates

Three layers, all required:

Pre-commit  → gitleaks / TruffleHog / detect-secrets pre-commit hook.
CI          → server-side scan on every PR; block merge on high-confidence finding.
Org policy  → push-protection at the SCM (GitHub Advanced Security secret scanning,
              GitLab Secret Detection); rotate any leaked secret immediately.

A leaked secret is rotated, not deleted. Git history retention defeats deletion.

Step 5 — Egress controls

Logs                → mask before write; CI runners must mark secrets as masked.
Stack traces        → never include secret values; sanitize at the boundary.
Error responses     → never echo the secret back, even on failure.
Telemetry / APM     → strip from request/response captures; allowlist headers.

Step 6 — Runtime guard: block before the secret lands

The scanning gates (Step 4) are the CI/pre-commit nets. The earliest gate is the agent itself: before it writes a credential into a tracked file or stages a commit, the secret-vcs-guard rule runs the secret_detector library and, on a hit, STOPS → shows the match (file:line · kind · masked · why) → asks via numbered options → offers the tiered alternative below. It never silently commits and never silently strips. VCS-agnostic: the same detector runs against the diff for git, SVN, and Mercurial (SVN/hg native hooks are server-side, a stronger but later net). The deterministic CI backstop is check_secret_leak — the agent gate is one layer, not the whole defense.

False positives — audited allowlist, never a global mute. Suppress a confirmed non-secret with an inline # secret-allow marker on the line, or a narrow entry (path or path:line) in a repo-root .secret-allow file. Each .secret-allow entry SHOULD carry a one-line justification comment (# …) so it is reviewable in the diff; the entry is line-scoped, so it cannot mute a new secret elsewhere in the same file.

Tiered alternative to suggest when a secret is caught (refines Step 1 for the in-the-moment fix; cite OWASP Secrets Management Cheat Sheet + CWE-798 + 12-factor):

Solo / local, dev-only        → gitignored .env + committed .env.example (keys, no values).
Team / production app         → cloud secret manager (AWS/GCP/Azure) or Vault / Doppler.
Kubernetes / GitOps           → SOPS (encrypt values in the committed file) or Sealed Secrets.
CI / deploy                   → OIDC federation to the cloud — no long-lived stored credential.

Remediation — a secret already reached VCS

Order matters; a history rewrite does not un-leak an already-pushed secret:

1. Rotate / revoke the credential now. Immediate, and the only step that actually stops the damage — assume it is compromised the moment it was pushed.
2. git rm / a deletion commit is insufficient — the secret stays in history (git log, blame, prior commits, clones, forks, CI logs).
3. Purge history with git filter-repo (preferred) or BFG Repo-Cleaner.
4. Force-push + coordinate re-clones; forks and existing clones still hold the old objects — which is why step 1 is the real fix.

Procedure: Apply to a new secret

1. Inspect the existing secret inventory and IaC for store conventions; run Step 1 and lock the store decision in code/IaC.
2. Define the access pattern (Step 2); choose static-vs-dynamic explicitly.
3. Write the rotation contract (Step 3) into the runbook before the secret ships.
4. Verify the three scanning gates (Step 4) cover the repo.
5. Audit egress paths (Step 5) for the new secret class.
6. Hand the design to a reviewer; cite this skill.

Output format

1. Secret-inventory entry: name · store · access-pattern · owner · rotation-period · mechanism.
2. Scanner-gate matrix: layer · tool · scope · failure mode.
3. Egress-control checklist with sign-off per category.

Gotcha

• "We rotate in Secrets Manager" — but the application caches the value forever. Cache TTL must be ≤ rotation grace.
• External Secrets Operator pulls into a k8s Secret; that Secret is base64, not encrypted. Threat-model node access accordingly.
• GitHub environment secrets are NOT available on pull_request events from forks — designs that rely on them silently break for external contributors.
• Vault dynamic creds expire faster than long-running connection pools assume; close + re-acquire on lease near-expiry, don't wait for the failure.
• Pre-commit scanners fire only when developers install the hook — CI scanners are the load-bearing gate.
• Agent/MCP config is a new leak surface. A raw provider key pasted into .mcp.json, an agent-config file, or a CI workflow YAML is a committed secret — the scanning gates (Step 4) must cover those paths too, and the config should reference ${ENV_VAR}, never the literal. An MCP server that ingests untrusted content and holds a live key is the egress leg of the lethal trifecta — see lethal-trifecta-guard.

Do NOT

• Do NOT commit a secret, even to a private repo. Rotate any leaked secret; deletion does not work.
• Do NOT pass secrets via CLI args (ps exposes them) — use env or stdin.
• Do NOT echo secrets in logs, stack traces, error responses, or APM captures.
• Do NOT hand-roll Kubernetes Secret objects committed to git — use External Secrets Operator.
• Do NOT inline the per-provider cookbooks into this skill — externalize per Sunset Policy.

Auto-trigger keywords

• secrets management
• secret rotation
• vault / aws secrets manager / azure key vault / gcp secret manager
• external secrets operator
• secret scanning / gitleaks / trufflehog
• credential leak

Provenance

• Adopted from: an external reference (internal provenance, redacted) — Sunset Policy applied: a large provider cookbook reduced to a ~140-line decision framework; per-provider deep-dives externalized to upstream docs below.
• Externalized provider docs:
  • HashiCorp Vault: https://developer.hashicorp.com/vault/docs · https://developer.hashicorp.com/vault/docs/secrets/databases
  • AWS Secrets Manager: https://docs.aws.amazon.com/secretsmanager/ · rotation: https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html
  • Azure Key Vault: https://learn.microsoft.com/en-us/azure/key-vault/general/
  • GCP Secret Manager: https://cloud.google.com/secret-manager/docs
  • External Secrets Operator: https://external-secrets.io/
  • GitHub secret scanning: https://docs.github.com/en/code-security/secret-scanning · gitleaks: https://github.com/gitleaks/gitleaks · TruffleHog: https://github.com/trufflesecurity/trufflehog
• Cross-linked: aws-infrastructure, security-audit, threat-modeling, security.
• Provenance registry: agents/settings/contexts/skills-provenance.yml (entry: secrets-management).
• Iron-Law floor: verify-before-complete, skill-quality, non-destructive-by-default.