Performing Kubernetes Etcd Security Assessment

Performing Kubernetes etcd Security Assessment

Overview

etcd is the distributed key-value store that serves as Kubernetes' backing store for all cluster data, including Secrets, RBAC policies, ConfigMaps, and workload configurations. Without proper hardening, etcd exposes all cluster secrets in plaintext, making it the highest-value target for attackers who gain control plane access. A comprehensive security assessment covers encryption at rest, TLS for transport, access control, backup security, and network isolation.

When to Use

• When conducting security assessments that involve performing kubernetes etcd security assessment
• When following incident response procedures for related security events
• When performing scheduled security testing or auditing activities
• When validating security controls through hands-on testing

Coverage Gaps & Validation

An etcd assessment that only reads API-server flags misses the cases that actually leak secrets:

• Encryption config present ≠ secrets encrypted: an EncryptionConfiguration with identity listed first, or applied after secrets were written, leaves existing secrets in plaintext. Don't trust the YAML - read a key straight from etcd and confirm the k8s:enc:aescbc:v1: prefix; plaintext key/value output means it's exposed.
• Listen address and client-cert auth: check --listen-client-urls/--advertise-client-urls aren't bound to 0.0.0.0, and that --client-cert-auth=true / --peer-client-cert-auth=true and --auto-tls=false are set - TLS being "on" without client-cert-auth still allows anonymous reads on 2379.
• Backups and peer port: snapshot files (snapshot save) are full plaintext-or-encrypted copies of every secret; an unencrypted backup defeats at-rest encryption. The peer port 2380 and member-to-member TLS are often forgotten.
• Network reachability: firewall rules for 2379/2380 must actually block worker nodes, not just be declared.

Validate by probing, not reading: dump a known Secret via etcdctl get /registry/secrets/... and inspect the bytes; from a worker node attempt curl -k https://<cp>:2379/health and confirm it's refused; verify cert expiry with openssl x509 -enddate; and snapshot-status a backup to prove integrity. Map findings to CIS 2.1-2.7 and re-check after key rotation that old keys were removed and secrets re-encrypted.

Prerequisites

• Access to Kubernetes control plane nodes
• SSH access to etcd cluster nodes (or etcdctl configured)
• CIS Kubernetes Benchmark reference document
• Understanding of TLS certificate management and EncryptionConfiguration

Assessment Areas

1. Encryption at Rest

Verify that Kubernetes encrypts Secret data stored in etcd:

# Check if EncryptionConfiguration is configured on API server
ps aux | grep kube-apiserver | grep encryption-provider-config

# View the encryption configuration
cat /etc/kubernetes/enc/encryption-config.yaml

Expected secure configuration:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}  # Fallback for reading unencrypted data

Verify secrets are actually encrypted in etcd:

# Read a secret directly from etcd
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/my-secret | hexdump -C | head -20

# If encrypted, output starts with "k8s:enc:aescbc:v1:key1"
# If NOT encrypted, you'll see plaintext key-value pairs

2. TLS Transport Security

# Verify etcd uses TLS for client connections
ETCDCTL_API=3 etcdctl endpoint health \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Check peer TLS configuration
ps aux | grep etcd | tr ' ' '\n' | grep -E "peer-cert|peer-key|peer-trusted-ca"

# Verify certificate expiration
openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -enddate
openssl x509 -in /etc/kubernetes/pki/etcd/peer.crt -noout -enddate

Expected flags:

3. Access Control

# Verify etcd is not exposed on all interfaces
ps aux | grep etcd | tr ' ' '\n' | grep listen-client-urls
# Should be: https://127.0.0.1:2379 (not 0.0.0.0)

# Check who can access etcd certificates
ls -la /etc/kubernetes/pki/etcd/
# Should be readable only by root/etcd user

# Verify API server is the only etcd client
ss -tlnp | grep 2379
# Only kube-apiserver should have connections

4. Backup Security

# Create an encrypted etcd backup
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Encrypt the backup file
gpg --symmetric --cipher-algo AES256 /backup/etcd-snapshot.db

# Verify backup integrity
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot.db --write-out=table

5. Network Isolation

# Verify etcd ports are firewalled
iptables -L -n | grep -E "2379|2380"

# Check if etcd is accessible from worker nodes (should NOT be)
# Run from a worker node:
curl -k https://<control-plane-ip>:2379/health
# Should be rejected/timeout

CIS Benchmark Checks

Key Rotation Procedure

# 1. Generate new encryption key
NEW_KEY=$(head -c 32 /dev/urandom | base64)

# 2. Update EncryptionConfiguration with new key first
cat > /etc/kubernetes/enc/encryption-config.yaml <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key2
              secret: ${NEW_KEY}
            - name: key1
              secret: <old-key>
      - identity: {}
EOF

# 3. Restart API server to pick up new config
# 4. Re-encrypt all secrets with new key
kubectl get secrets --all-namespaces -o json | \
  kubectl replace -f -

# 5. Remove old key from EncryptionConfiguration
# 6. Restart API server again

References

• Kubernetes etcd Encryption Documentation
• CIS Kubernetes Benchmark - etcd Controls
• Securing etcd - K8s Security Guide
• Infosec: Encryption and etcd