Secrets Management

Secrets Management Rules

1. Secret Types and Classification

Classification by Sensitivity

Classification Rules

• All secrets must be classified before storage
• Classification determines storage location, rotation policy, and access control
• When in doubt, classify higher — downgrade after review

2. Storage Solutions Comparison

Selection Criteria

• Multi-cloud or vendor-neutral: HashiCorp Vault
• Single cloud, managed: Use cloud provider's secret manager
• Kubernetes-native, simple: External Secrets Operator + cloud SM
• GitOps with encryption: SOPS with Age/KMS
• Dynamic credentials needed: Vault (database, cloud IAM)

3. Rotation Policies

Automated Rotation Pattern

1. Generate new secret (Version N+1)
2. Deploy new secret to consumers (dual-credential phase)
3. Verify consumers use Version N+1
4. Revoke old secret (Version N)
5. Remove old secret from storage

Zero-Downtime Rotation (Dual-Credential)

Phase 1: [Active: V1]
Phase 2: [Active: V1, V2]  ← Deploy V2, app accepts both
Phase 3: [Active: V2]       ← Remove V1 from app config
Phase 4: [Active: V2]       ← Revoke V1 credential

Rotation Frequency by Type

Emergency Rotation

Trigger immediate rotation when:

• Secret confirmed or suspected leaked
• Team member with access leaves the organization
• Unauthorized access detected in audit logs
• Compliance audit finding

Emergency Rotation Procedure

1. IMMEDIATE (0-15 min)
   - Identify all systems using the compromised secret
   - Generate new secret (Version N+1)
   - Deploy new secret to all consumers

2. CONTAIN (15-60 min)
   - Revoke the compromised secret (Version N)
   - Verify all consumers are using Version N+1
   - Check audit logs for unauthorized usage during exposure window

3. ASSESS (1-4 hours)
   - Determine exposure timeline (when leaked, when detected)
   - Identify blast radius (what data/systems were accessible)
   - Document findings for incident report

4. REMEDIATE (1-7 days)
   - Conduct root cause analysis (how the secret was leaked)
   - Implement prevention measures (pre-commit hooks, secret scanning)
   - Update runbooks and rotation procedures
   - File incident report per incident-response process

4. Kubernetes Secret Patterns

External Secrets Operator (ESO)

# ClusterSecretStore — connect to AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets
---
# ExternalSecret — sync a specific secret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-db-credentials
  namespace: myapp
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: myapp-db-credentials
    creationPolicy: Owner
    deletionPolicy: Retain
  data:
    - secretKey: username
      remoteRef:
        key: myapp/prod/database
        property: username
    - secretKey: password
      remoteRef:
        key: myapp/prod/database
        property: password

See references/kubernetes-patterns.md for Sealed Secrets and CSI Secret Store Driver examples.

Pattern Selection Guide

5. Application Integration

Secret Consumption Patterns

See references/kubernetes-patterns.md for K8s environment variable and file mount configuration examples.

Application-Level Rules

• Never log secret values — mask in log output
• Cache secrets in memory with TTL, not on disk
• Handle secret refresh without restart when possible
• Use connection pooling that supports credential rotation
• Fail closed — refuse to start if required secrets are unavailable

6. Access Control

Least Privilege Principles

Dynamic Secrets (Vault)

1. App authenticates to Vault (K8s service account, AWS IAM)
2. App requests database credentials
3. Vault creates temporary DB user with limited permissions
4. Credential has TTL (e.g., 1 hour)
5. Vault revokes credential on TTL expiry or app shutdown

Advantages of Dynamic Secrets

• No shared credentials — each app instance gets unique credentials
• Automatic revocation — no manual cleanup
• Audit trail — every credential issuance is logged
• Blast radius reduction — compromised credential has limited scope and lifetime

7. CI/CD Pipeline Secrets

GitHub Actions

# Use OIDC for keyless cloud authentication (preferred)
permissions:
  id-token: write
  contents: read

steps:
  - name: Configure AWS credentials
    uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789:role/github-actions
      aws-region: us-east-1

  # Use GitHub Secrets for non-cloud credentials
  - name: Deploy
    env:
      API_KEY: ${{ secrets.DEPLOY_API_KEY }}
    run: ./deploy.sh

Pipeline Secret Rules

Preventing Secret Leakage in CI

# GitHub Actions — mask any dynamic secret
- name: Mask secret
  run: echo "::add-mask::${{ steps.get-secret.outputs.value }}"

# Prevent secrets in artifacts
- name: Build
  run: |
    # Never write secrets to files that become artifacts
    export DB_URL="${{ secrets.DB_URL }}"
    ./build.sh

8. Certificate Management

cert-manager with Let's Encrypt

# ClusterIssuer for automatic certificate provisioning
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
      - http01:
          ingress:
            class: nginx
      - dns01:
          route53:
            region: us-east-1
---
# Certificate request
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: myapp-tls
  namespace: myapp
spec:
  secretName: myapp-tls-secret
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - myapp.example.com
    - "*.myapp.example.com"
  renewBefore: 360h  # Renew 15 days before expiry

Certificate Lifecycle

mTLS Between Services

• Use cert-manager for issuing client certificates
• Alternatively, use service mesh (Istio, Linkerd) for automatic mTLS
• Internal CA for service-to-service communication
• Rotate internal CA annually, service certs every 90 days

9. Secret Detection and Prevention

Pre-Commit Hooks

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.21.2
    hooks:
      - id: gitleaks

  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.5.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']

CI Pipeline Scanning

# GitHub Actions
- name: Scan for secrets
  uses: gitleaks/gitleaks-action@v2
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Tool Comparison

Remediation When Secrets Leak

1. IMMEDIATE: Rotate the compromised secret
2. Assess: Determine exposure window and blast radius
3. Audit: Check access logs for unauthorized usage
4. Clean: Remove from Git history (BFG or git-filter-repo)
5. Prevent: Add detection to pre-commit and CI
6. Document: Record in incident log

# Remove secret from Git history
git filter-repo --invert-paths --path config/secrets.yaml

# Or use BFG Repo Cleaner
bfg --replace-text passwords.txt repo.git

Prevention Rules

• Enable pre-commit hooks for all repositories
• Run secret scanning in CI as a blocking check
• Maintain a .secrets.baseline file for known false positives
• Review .gitignore for secret-prone patterns (.env, *.pem, *.key)
• Train developers on secret hygiene during onboarding

10. Anti-Patterns

• Storing secrets in plaintext in Git — even in "private" repositories
• Using the same secret across environments — compromising dev exposes production
• Manual secret rotation without documentation — leads to outages during rotation
• Hardcoding secrets in application code — impossible to rotate without redeployment
• Sharing personal credentials for service access — no audit trail, no revocation control
• Long-lived static credentials without rotation — increases window of exposure
• Storing secrets in ConfigMaps instead of Secrets — no base64 encoding, no RBAC distinction
• No audit logging for secret access — cannot detect unauthorized usage
• Using root/admin credentials in applications — violates least privilege
• Embedding secrets in Docker images — exposed via docker history or image scanning

Additional References

• For HashiCorp Vault integration, setup, policies, and dynamic secrets, see references/vault-integration.md

Related Skills

• For incident response procedures when secrets are leaked, see incident-response skill
• For Kubernetes secret handling in manifests, see k8s-workflow skill
• For application-level security rules, see security skill