Security Vault Patterns

Vault Patterns

Centralize secrets in a vault, issue dynamic short-lived credentials, encrypt data through a transit engine, and eliminate long-lived secrets from your infrastructure

When to Use

• Centralizing secrets management across multiple services and environments
• Replacing long-lived database credentials with dynamic, short-lived ones
• Implementing envelope encryption for data at rest
• Managing TLS certificates and PKI infrastructure
• Designing secrets access patterns for Kubernetes workloads
• Evaluating HashiCorp Vault vs cloud-native KMS (AWS KMS, GCP KMS, Azure Key Vault)

Threat Context

Static, long-lived credentials are the most exploited vulnerability in cloud environments. The 2019 Capital One breach used stolen IAM credentials that had overly broad access and never rotated -- a single misconfigured WAF rule exposed temporary credentials that could access S3 buckets containing 106 million customer records. AWS reports that exposed access keys are exploited within minutes of being pushed to public repositories (automated bots scan GitHub continuously). The 2021 Codecov breach exfiltrated credentials from CI/CD environment variables. Dynamic credentials that expire in minutes instead of lasting forever reduce the attack window from "indefinitely" to "minutes." Vault systems also provide audit trails for every secret access, enabling detection of unauthorized access patterns that static credential storage cannot provide.

Instructions

1. Choose the right vault architecture. HashiCorp Vault: self-hosted or HCP Vault (managed), richest feature set, steepest learning curve. AWS Secrets Manager + KMS: cloud-native, simple, integrated with IAM. GCP Secret Manager + Cloud KMS: similar to AWS, integrated with Workload Identity. Azure Key Vault: similar to AWS, integrated with Managed Identity. Sealed Secrets (Bitnami): Kubernetes-specific, encrypts secrets for GitOps workflows. For multi-cloud or on-premise: HashiCorp Vault. For single-cloud with simple requirements: use the native secrets manager.

2. Use dynamic secrets instead of static credentials. Vault's database secret engine generates unique database credentials per request with configurable TTL (e.g., 1 hour). When the TTL expires, the credentials are automatically revoked. If a credential is compromised, the blast radius is limited to that credential's TTL and that single consumer. No more shared database passwords in config files. Dynamic secrets exist for databases (PostgreSQL, MySQL, MongoDB), cloud providers (AWS, GCP, Azure IAM), SSH certificates, and PKI certificates.

3. Implement envelope encryption for data at rest. Do not store the data encryption key (DEK) alongside the data. Encrypt the DEK with a key encryption key (KEK) managed by the vault's transit engine or KMS. Store the encrypted DEK with the data. To decrypt: send the encrypted DEK to the vault, get back the plaintext DEK, decrypt the data locally. This means the vault never sees your data -- it only manages keys. This pattern is used by AWS S3 server-side encryption, Google Cloud Storage, and Azure Storage encryption.

4. Authenticate workloads to the vault. Kubernetes: use the Kubernetes auth method (service account token authentication). AWS: use the IAM auth method (instance profile or IAM role). AppRole: for CI/CD and automated systems (role ID + secret ID). Never hardcode vault tokens. Every workload authenticates with the identity it already has (Kubernetes service account, AWS IAM role, etc.). The authentication method proves the workload's identity without introducing new credentials.

5. Implement PKI and certificate management. Vault's PKI secret engine acts as a certificate authority. Issue short-lived TLS certificates (24-72 hours) to services. Short-lived certificates eliminate the need for revocation lists (CRLs) because they expire before an attacker can meaningfully exploit a compromised certificate. Automate certificate issuance and renewal via cert-manager (Kubernetes) or Vault Agent.

6. Seal and unseal operations. HashiCorp Vault uses a seal mechanism: the master key is split into shares (Shamir's Secret Sharing). A threshold of shares (e.g., 3 of 5) is required to unseal the vault. In production, use auto-unseal with a cloud KMS (the KMS key unseals the vault master key). This eliminates the need for human operators to provide key shares during restarts while maintaining the security property that the vault's data is encrypted at rest.

Details

Dynamic Secrets Lifecycle

The lifecycle of a dynamic database credential: application authenticates to Vault with its Kubernetes service account token, requests database credentials from the database secret engine, Vault creates a unique database user with the required grants and a 1-hour TTL, Vault returns the username and password, application uses them for database connections, TTL expires, Vault revokes the database user by executing DROP USER. Compare this to static credentials: created once by a human, shared across services via environment variables or config files, never expire, never rotated, compromise is permanent and undetectable.

Transit Engine for Application-Level Encryption

The transit engine performs encrypt/decrypt operations without exposing the key material. The API is simple: POST /transit/encrypt/my-key with plaintext data returns ciphertext; POST /transit/decrypt/my-key with ciphertext returns plaintext. Key rotation is transparent: new versions of the key are created, old ciphertexts can still be decrypted, re-encryption can be done in batches using POST /transit/rewrap/my-key. This allows key rotation without downtime and without re-encrypting all data immediately.

Vault in Kubernetes

Three integration patterns: Vault Agent Injector (sidecar that fetches secrets and writes them to a shared volume as files, best for dynamic secrets that need refresh), CSI Secret Store Driver (mounts secrets as files via the Container Storage Interface, simpler but no automatic refresh), External Secrets Operator (syncs vault secrets to Kubernetes Secret objects, best for static secrets that change infrequently). Recommendation: Vault Agent for workloads that need dynamic secrets with automatic renewal, External Secrets Operator for simpler static secret synchronization.

Disaster Recovery

Vault contains all your secrets -- losing it is catastrophic. Implement: regular automated snapshots (Vault snapshot agent), cross-region replication (Vault Enterprise performance replication), a DR cluster (Vault Enterprise DR replication with automated failover), and tested restore procedures. Test recovery quarterly by restoring a snapshot to a staging environment and verifying all secret engines are functional. Document the recovery procedure so it can be executed under pressure during an actual incident.

Vault vs Cloud-Native KMS Decision Matrix

For teams with dedicated platform engineering, Vault provides the richest feature set. For small teams on a single cloud, the cloud-native option is operationally simpler.

Anti-Patterns

1. Vault with a single unseal key. If one person has the unseal key and they are unavailable, no one can unseal the vault after a restart. Use Shamir's Secret Sharing with a 3-of-5 (or similar) threshold, or auto-unseal with a cloud KMS. Single points of failure in secrets infrastructure are unacceptable.

2. Long-lived vault tokens. Vault tokens with no TTL or very long TTLs defeat the purpose of centralized secrets management. If a token is stolen, it provides indefinite access. Use short-lived tokens (1-24 hours) with renewal, and configure max TTLs on token roles so that even renewed tokens eventually expire.

3. Vault as a dumb key-value store. Using Vault only to store static secrets misses its most powerful feature: dynamic secrets. If you are storing a database password in Vault and sharing it across 10 services, you have centralized the static credential but have not eliminated it. Use dynamic secrets to generate unique, short-lived credentials per consumer.

4. No audit logging. Vault has a built-in audit device that logs every operation including the accessor, timestamp, and path accessed. Failing to enable it means you cannot detect unauthorized secret access, compromised tokens, or policy violations. Enable audit logging to at least one persistent backend (syslog, file, socket).

5. Skipping seal/unseal understanding. Deploying Vault without understanding the seal mechanism leads to outages when the vault is sealed (due to restart, crash, or manual seal) and no one knows how to unseal it. Document the unseal procedure, test it during incident drills, and ensure multiple team members can perform it.