DevOps Secrets
Agent Workflow (MANDATORY)
Before ANY change to a secret store, a secret manifest or a rotation, spawn in parallel, one spawn_agent call each:
- explore-codebase - find
.sops.yaml, SealedSecret/ExternalSecret manifests, Vault/OpenBao config and policies, CI secret references,.gitleaks.toml, existing rotation runbooks - research-expert - verify flags, API versions and licence/support status at the official source (developer.hashicorp.com/vault, openbao.org, getsops.io, external-secrets.io, github.com/bitnami/sealed-secrets)
- mcp__context7__query-docs -
/websites/developer_hashicorp_vault,/getsops/sops,/websites/external-secrets_io,/bitnami/sealed-secrets
Then inspect the target read-only (vault status, kubectl get externalsecret -A, git diff --stat on encrypted files; never a decrypt to stdout). After writing files in a repo, run sniper.
Versions (checked 2026-11-10)
Vault v2.2.1, OpenBao v2.7.2, SOPS v3.13.3, age v1.3.2, Sealed Secrets v0.41.2, External Secrets Operator v2.15.0, gitleaks v8.30.1, trufflehog v3.99.8 (GitHub releases/latest redirects). Vault source licence: Business Source License 1.1 (licensor IBM, change licence MPL 2.0 after four years, internal use not a competitive offering); OpenBao is hosted by the Linux Foundation as "a Series of LF Projects" (its licence file was not read). ESO supports only the current minor release. gitleaks states it is feature complete with security patches only. Details and URLs: each reference's Sources.
Pinned versions:
vault: "2.2.1 (BUSL 1.1 licence, IBM licensor)"
openbao: "2.7.2 (docs 2.7.x, a Series of LF Projects)"
sops: "3.13.3"
age: "1.3.2"
sealed-secrets: "0.41.2 (repo bitnami/sealed-secrets)"
external-secrets: "2.15.0 (only the newest minor is supported)"
gitleaks: "8.30.1 (feature complete, security patches only)"
trufflehog: "3.99.8"
Critical Rules
- Never print a secret value. Not in chat, logs, diffs,
describe,-o yamlof a Secret,sops -dto stdout,vault kv getwithout-fieldinto a pipe you cannot see, or avault readof dynamic credentials. Show names, keys, paths, versions, fingerprints, lease IDs and timestamps only. Redact before quoting anything. - Never put a secret value on a command line. Arguments land in shell history,
psand CI logs. Read values from a file with mode 0600, stdin, a prompt or an environment variable set by the secret store. - A leaked secret is revoked and rotated first. Rewriting git history is cleanup, never remediation (leak-prevention-response.md).
- Prefer short-lived and identity-based over static. Dynamic credentials, OIDC federation and workload identity beat long-lived keys in Git, CI variables or Secrets.
- Destructive or irreversible secret operations are proposed, not run. Use the ACTION / EFFECT / PROOF / WAIT-FOR-GO format and state what becomes unrecoverable (destructive-secret-ops.md).
- Key custody before anything else. A key whose only copy sits where the data sits (unseal keys, age identity, Sealed Secrets controller key, encryption-at-rest key) is a lost-data incident waiting; confirm an offline copy exists before encrypting at scale.
- Never invent a flag, CRD field or API. Cite the doc page; unknown = "not verified".
Overview
| Need | Pick | Why in one line |
|---|---|---|
| Few secrets, GitOps, no server to run | SOPS + age | Encrypted files in Git, one private key to guard |
| Kubernetes-only, write-only Git workflow, small team | Sealed Secrets | Public-key encryption, controller decrypts in-cluster |
| Many secrets, one source of truth, rotation, audit | Vault or OpenBao + ESO | Dynamic credentials, leases, policies, audit devices |
| Per-request database/PKI credentials | Vault/OpenBao engines | Unique short-lived credentials, automatic revocation |
| Public-cloud secret manager already in use | ESO provider | One-line pointer; deep dive out of scope |
| Build-time secret | BuildKit secret mount | Never ARG/ENV (container-ci-ansible-secrets.md) |
| CI to infrastructure | OIDC federation | No static key to leak |
Architecture
repo/
├── .sops.yaml # creation rules: path regex -> age recipients
├── secrets/ # SOPS-encrypted files only (values ciphertext)
├── k8s/
│ ├── secretstore.yaml # ESO SecretStore / ClusterSecretStore
│ ├── externalsecret.yaml # what to fetch; the cluster builds the Secret
│ └── sealedsecret.yaml # or: ciphertext sealed for one cluster
├── vault/policies/*.hcl # least-privilege policies, reviewed like code
├── .gitleaks.toml # extends the default rules
└── runbooks/rotation-<secret>.md
Reference Guide
Concepts
| Topic | Reference | When to Consult |
|---|---|---|
| Lifecycle and tool choice | secrets-lifecycle.md | New secret, tool decision, bootstrap problem, agent output rules |
| Vault/OpenBao core | vault-openbao-core.md | Status and licence, seal/unseal, auto-unseal, raft, audit devices |
| Vault/OpenBao access | vault-openbao-access.md | Auth methods, policies, KV v2, dynamic DB credentials, PKI |
| SOPS + age | sops-age.md | Encrypted files in Git, key custody, recipients |
| Sealed Secrets | sealed-secrets.md | Scopes, controller key backup/renewal |
| External Secrets Operator | external-secrets-operator.md | SecretStore, ExternalSecret, multi-tenancy |
| Kubernetes Secrets | kubernetes-secrets-at-rest.md | base64, EncryptionConfiguration, KMS, RBAC |
| Docker, CI, Ansible | container-ci-ansible-secrets.md | Compose secrets, BuildKit, OIDC |
| Machine identity and certs | machine-identity-certificates.md | Workload identity, SPIFFE pointer, certificates |
| Leaks | leak-prevention-response.md | Scanners, pre-commit, response order |
| Rotation | rotation-lifecycle.md | Rotation patterns per secret type |
| Destructive operations | destructive-secret-ops.md | Any irreversible secret command |
Templates
| Template | When to Use |
|---|---|
| sops.yaml | Copy as .sops.yaml; age recipients per path |
| external-secrets.yaml | SecretStore (Vault, Kubernetes auth) + ExternalSecret + ClusterSecretStore |
| sealed-secret-workflow.md | Seal a Secret offline, back up the controller key |
| vault-policy.hcl | Least-privilege KV v2 + dynamic DB policy |
| vault-kubernetes-auth-role.sh | Enable Kubernetes auth, define a role (reference only, not executable as is) |
| encryption-configuration.yaml | kube-apiserver encryption at rest |
| gitleaks.toml | Extend default rules, scoped allowlist |
| leaked-secret-response.md | Incident checklist |
| rotation-runbook.md | Per-secret runbook skeleton |
Quick Reference
vault status # sealed? HA? version (read-only, no auth needed)
vault kv metadata get -mount=secret app/db # versions and timestamps, no value
vault lease lookup <lease-id> # lease TTL, no value
kubectl --context "$CTX" -n "$NS" get externalsecret # READY / STATUS, no value
kubectl --context "$CTX" -n "$NS" get secret db-credentials -o jsonpath='{.data}' | jq 'keys'
gitleaks dir --redact --no-banner . # scan working tree, secrets masked in output
gitleaks git --redact --no-banner . # scan history, secrets masked
Best Practices
DO
- Write a rotation runbook before the first production use of any secret
- Keep the unseal/recovery keys, age identities and Sealed Secrets key backups in separate custody, tested by a restore
- Enable at least two Vault/OpenBao audit devices so one failing does not stop the service
- Scope ESO
ClusterSecretStoreaccess withconditionsand give each store the least-privilege role
DON'T
- Treat base64 or a private Git repo as protection
- Commit
.env, kubeconfig, age identity files or Terraform state holding secrets - Rely on git history rewriting after a leak
- Disable the pre-commit scanner with
SKIP=as a habit
Related skills
- devops-kubernetes
- devops-k8s-cluster
- devops-gitops-packaging
- devops-cicd
- devops-docker
- devops-ansible
- devops-networking
- devops-linux
- devops-storage-backup
- devops-sre-ops