Implementing Secrets Management With Vault

Implementing Secrets Management with Vault

When to Use

• When applications store database passwords, API keys, or certificates in environment variables or config files
• When migrating from static long-lived credentials to dynamic short-lived secrets
• When Kubernetes workloads need secure access to database credentials or cloud provider APIs
• When compliance requirements mandate centralized credential management with audit logging
• When CI/CD pipelines contain hardcoded secrets that represent supply chain risk

Do not use for AWS-only environments where AWS Secrets Manager suffices without multi-cloud requirements, for application-level encryption logic (though Vault Transit can help), or for identity federation (see managing-cloud-identity-with-okta).

Common Misconfigurations & Verification

• Root token not revoked after init: the initial root token from vault operator init stays valid forever if left around. Confirm vault token lookup <root-token> errors out; revoke with vault token revoke -self and rely on OIDC/AppRole for ongoing access.
• Root DB credentials never rotated: if you skip vault write -force database/rotate-root/<db>, the bootstrap password in connection_url still works outside Vault. Verify by attempting a direct login with the original password - it must fail.
• secret_id_num_uses=0 / secret_id_ttl=0: unlimited, non-expiring AppRole SecretIDs defeat short-lived auth. Audit with vault read auth/approle/role/<name> and flag any 0 values.
• Audit device disabled or single sink: if vault audit list is empty, secret access is unlogged. Enable at least one file plus one syslog/socket device so a single sink outage doesn't silence the trail.
• Overbroad policies: a policy missing the path "sys/*" { capabilities = ["deny"] } rule or granting sudo leaks admin paths. Run vault policy read <name>, then vault token create -policy=<name> and probe the paths that should be denied.
• TTLs longer than the workload: dynamic DB creds with default_ttl=24h outlive most jobs. Keep TTLs near actual usage and confirm a test lease auto-revokes with vault lease lookup/vault lease revoke.
• Kubernetes auth pinned to a long-lived reviewer JWT: prefer the short-lived projected SA token; verify auth/kubernetes/config does not hardcode a non-expiring token_reviewer_jwt.
• Don't declare it secured until the root token is revoked, every root/static credential Vault now manages is rotated, audit logging survives a sink failure, and a test lease is shown to auto-revoke at TTL.

Prerequisites

• HashiCorp Vault server deployed in HA mode (Consul or Raft storage backend)
• TLS certificates for Vault listener endpoints
• Vault Enterprise license for namespaces, Sentinel policies, and replication (optional)
• Kubernetes cluster with Vault Agent Injector or CSI provider for workload integration

Workflow

Step 1: Deploy Vault in High Availability Mode

Deploy Vault using Integrated Storage (Raft) for HA without external dependencies. Configure TLS, audit logging, and auto-unseal using a cloud KMS.

# vault-config.hcl
storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-node-1"

  retry_join {
    leader_api_addr = "https://vault-node-2.internal:8200"
  }
  retry_join {
    leader_api_addr = "https://vault-node-3.internal:8200"
  }
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/vault.crt"
  tls_key_file  = "/opt/vault/tls/vault.key"
}

seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "alias/vault-unseal-key"
}

api_addr      = "https://vault-node-1.internal:8200"
cluster_addr  = "https://vault-node-1.internal:8201"

telemetry {
  prometheus_retention_time = "30s"
  disable_hostname         = true
}

# Initialize Vault
vault operator init -key-shares=5 -key-threshold=3

# Enable audit logging
vault audit enable file file_path=/var/log/vault/audit.log

# Enable syslog audit for SIEM integration
vault audit enable syslog tag="vault" facility="AUTH"

Step 2: Configure Authentication Methods

Enable authentication backends for human operators, applications, and CI/CD pipelines. Use AppRole for machine authentication and OIDC for human access.

# Enable OIDC auth for human users via Okta
vault auth enable oidc
vault write auth/oidc/config \
  oidc_discovery_url="https://company.okta.com/oauth2/default" \
  oidc_client_id="vault-client-id" \
  oidc_client_secret="vault-client-secret" \
  default_role="default"

# Enable AppRole for application authentication
vault auth enable approle
vault write auth/approle/role/web-app \
  secret_id_ttl=10m \
  token_num_uses=10 \
  token_ttl=20m \
  token_max_ttl=30m \
  secret_id_num_uses=1 \
  token_policies="web-app-policy"

# Enable Kubernetes auth for pod-based access
vault auth enable kubernetes
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443" \
  token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Step 3: Enable Dynamic Secret Engines

Configure database secret engines to generate short-lived credentials on demand. Each credential set has a TTL and is automatically revoked when it expires.

# Enable database secrets engine for PostgreSQL
vault secrets enable database
vault write database/config/production-db \
  plugin_name=postgresql-database-plugin \
  allowed_roles="readonly,readwrite" \
  connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/production?sslmode=require" \
  username="vault_admin" \
  password="initial-password"

# Rotate the root credentials so Vault manages them exclusively
vault write -force database/rotate-root/production-db

# Create a readonly role with 1-hour TTL
vault write database/roles/readonly \
  db_name=production-db \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  revocation_statements="REVOKE ALL ON ALL TABLES IN SCHEMA public FROM \"{{name}}\"; DROP ROLE IF EXISTS \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Enable AWS secrets engine for dynamic IAM credentials
vault secrets enable aws
vault write aws/config/root \
  access_key=AKIAEXAMPLE \
  secret_key=secretkey \
  region=us-east-1

vault write aws/roles/deploy-role \
  credential_type=iam_user \
  policy_document=@deploy-policy.json \
  default_sts_ttl=3600

Step 4: Integrate with Kubernetes Workloads

Use the Vault Agent Injector or CSI Provider to deliver secrets to pods without application code changes. Secrets are rendered as files in a shared volume.

# Kubernetes deployment with Vault Agent Injector annotations
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "web-app"
        vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/readonly"
        vault.hashicorp.com/agent-inject-template-db-creds: |
          {{- with secret "database/creds/readonly" -}}
          export DB_USERNAME="{{ .Data.username }}"
          export DB_PASSWORD="{{ .Data.password }}"
          {{- end }}
    spec:
      serviceAccountName: web-app
      containers:
        - name: web-app
          image: company/web-app:v2.1
          command: ["/bin/sh", "-c", "source /vault/secrets/db-creds && ./start.sh"]

Step 5: Implement Transit Encryption and PKI

Use the Transit secrets engine for application-level encryption without managing keys in application code. Deploy the PKI engine for automatic TLS certificate management.

# Enable Transit engine for encryption as a service
vault secrets enable transit
vault write -f transit/keys/payment-data type=aes256-gcm96

# Encrypt sensitive data
vault write transit/encrypt/payment-data \
  plaintext=$(echo "card-number-4111-1111-1111-1111" | base64)

# Enable PKI for internal certificate management
vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki

# Generate root CA
vault write pki/root/generate/internal \
  common_name="Internal Root CA" \
  ttl=87600h

# Configure intermediate CA for issuing certificates
vault secrets enable -path=pki_int pki
vault write pki_int/intermediate/generate/internal \
  common_name="Internal Intermediate CA" \
  ttl=43800h

# Create a role for issuing certificates
vault write pki_int/roles/internal-services \
  allowed_domains="internal.company.com" \
  allow_subdomains=true \
  max_ttl=720h

Step 6: Establish Policies and Audit Trail

Define fine-grained ACL policies following least privilege. Enable comprehensive audit logging for all secret access and administrative operations.

# web-app-policy.hcl
path "database/creds/readonly" {
  capabilities = ["read"]
}

path "transit/encrypt/payment-data" {
  capabilities = ["update"]
}

path "transit/decrypt/payment-data" {
  capabilities = ["update"]
}

path "secret/data/web-app/*" {
  capabilities = ["read", "list"]
}

# Deny access to admin paths
path "sys/*" {
  capabilities = ["deny"]
}

# Apply the policy
vault policy write web-app-policy web-app-policy.hcl

# Verify audit log captures all operations
vault audit list -detailed

Key Concepts

Tools & Systems

• HashiCorp Vault: Core secrets management platform providing dynamic secrets, encryption, and identity-based access
• Vault Agent Injector: Kubernetes mutating webhook that automatically injects Vault secrets into pod volumes via sidecar containers
• Vault CSI Provider: Kubernetes CSI driver that mounts Vault secrets directly into pod volumes without sidecar containers
• consul-template: Template rendering daemon that watches Vault secrets and re-renders configuration files when secrets change
• Vault Radar: Secret scanning tool that detects hardcoded credentials in source code, CI/CD pipelines, and cloud configurations

Common Scenarios

Scenario: Eliminating Hardcoded Database Credentials from CI/CD Pipeline

Context: A DevOps team stores PostgreSQL credentials in GitHub Actions secrets and Jenkins credential stores. The same credentials are shared across staging and production environments with no rotation for 18 months.

Approach:

1. Deploy Vault with AppRole auth enabled for CI/CD systems
2. Configure the database secrets engine with separate roles for staging (readwrite, 2h TTL) and production (readonly, 1h TTL)
3. Create separate Vault policies for each pipeline stage restricting access to the appropriate database role
4. Update GitHub Actions workflows to authenticate via AppRole and request dynamic credentials at the start of each job
5. Rotate the static PostgreSQL credentials and hand root access to Vault exclusively
6. Enable audit logging to track every credential request with pipeline job metadata

Pitfalls: Failing to rotate the original static credentials after Vault migration leaves the old credentials valid. Setting TTLs too short causes credential expiry mid-deployment for long-running jobs.

Output Format

Vault Secrets Management Audit Report
=======================================
Vault Cluster: vault.internal.company.com
Version: 1.18.1 Enterprise
HA Mode: Raft (3 nodes)
Seal Type: AWS KMS Auto-Unseal
Report Date: 2025-02-23

SECRET ENGINES:
  database/         PostgreSQL dynamic creds   Leases Active: 47
  aws/              Dynamic IAM credentials    Leases Active: 12
  transit/          Encryption as a service    Keys: 8
  pki/              Root CA                    Certs Issued: 0
  pki_int/          Intermediate CA            Certs Issued: 234
  secret/           KV v2 static secrets       Versions: 1,892

AUTH METHODS:
  oidc/             Okta SSO for humans        Active Tokens: 23
  approle/          CI/CD pipelines            Active Tokens: 156
  kubernetes/       Pod-based auth             Active Tokens: 89

AUDIT FINDINGS:
  [WARN] 3 AppRole secret_id_num_uses set to 0 (unlimited)
  [WARN] 12 KV secrets not accessed in 90+ days (potential orphans)
  [PASS] All dynamic secret TTLs under 24 hours
  [PASS] Audit logging enabled on all nodes
  [PASS] Root token revoked after initial setup

CREDENTIAL HYGIENE:
  Static Secrets (KV): 234
  Dynamic Secrets Active: 59
  Average Lease TTL: 2.3 hours
  Secrets Rotated This Month: 12,456