Performing Kubernetes Etcd Security Assessment

Performing Kubernetes etcd Security Assessment

Overview

etcd is the distributed key-value store that serves as Kubernetes' backing store for all cluster data, including Secrets, RBAC policies, ConfigMaps, and workload configurations. Without proper hardening, etcd exposes all cluster secrets in plaintext, making it the highest-value target for attackers who gain control plane access. A comprehensive security assessment covers encryption at rest, TLS for transport, access control, backup security, and network isolation.

When to Use

Trigger phrases:

• "performing kubernetes etcd security assessment"

• "Assess the security posture of Kubernetes etcd clusters by evaluating encryption"

• When conducting security assessments that involve performing kubernetes etcd security assessment

• When following incident response procedures for related security events

• When performing scheduled security testing or auditing activities

• When validating security controls through hands-on testing

Prerequisites

• Access to Kubernetes control plane nodes
• SSH access to etcd cluster nodes (or etcdctl configured)
• CIS Kubernetes Benchmark reference document
• Understanding of TLS certificate management and EncryptionConfiguration

Assessment Areas

This section covers assessment areas for performing kubernetes etcd security assessment.

• Ensure all prerequisites are met before proceeding
• Follow the documented workflow steps in sequence
• Record results and any anomalies encountered during this phase

1. Encryption at Rest

Verify that Kubernetes encrypts Secret data stored in etcd:

# Check if EncryptionConfiguration is configured on API server
ps aux | grep kube-apiserver | grep encryption-provider-config

# View the encryption configuration
cat /etc/kubernetes/enc/encryption-config.yaml

Expected secure configuration:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}  # Fallback for reading unencrypted data

Verify secrets are actually encrypted in etcd:

# Read a secret directly from etcd
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/my-secret | hexdump -C | head -20

# If encrypted, output starts with "k8s:enc:aescbc:v1:key1"
# If NOT encrypted, you'll see plaintext key-value pairs

2. TLS Transport Security

# Verify etcd uses TLS for client connections
ETCDCTL_API=3 etcdctl endpoint health \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Check peer TLS configuration
ps aux | grep etcd | tr ' ' '\n' | grep -E "peer-cert|peer-key|peer-trusted-ca"

# Verify certificate expiration
openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -enddate
openssl x509 -in /etc/kubernetes/pki/etcd/peer.crt -noout -enddate

Expected flags:

3. Access Control

# Verify etcd is not exposed on all interfaces
ps aux | grep etcd | tr ' ' '\n' | grep listen-client-urls
# Should be: https://127.0.0.1:2379 (not 0.0.0.0)

# Check who can access etcd certificates
ls -la /etc/kubernetes/pki/etcd/
# Should be readable only by root/etcd user

# Verify API server is the only etcd client
ss -tlnp | grep 2379
# Only kube-apiserver should have connections

4. Backup Security

# Create an encrypted etcd backup
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Encrypt the backup file
gpg --symmetric --cipher-algo AES256 /backup/etcd-snapshot.db

# Verify backup integrity
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot.db --write-out=table

5. Network Isolation

# Verify etcd ports are firewalled
iptables -L -n | grep -E "2379|2380"

# Check if etcd is accessible from worker nodes (should NOT be)
# Run from a worker node:
curl -k https://<control-plane-ip>:2379/health
# Should be rejected/timeout

CIS Benchmark Checks

Key Rotation Procedure

# 1. Generate new encryption key
NEW_KEY=$(head -c 32 /dev/urandom | base64)

# 2. Update EncryptionConfiguration with new key first
cat > /etc/kubernetes/enc/encryption-config.yaml <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key2
              secret: ${NEW_KEY}
            - name: key1
              secret: <old-key>
      - identity: {}
EOF

# 3. Restart API server to pick up new config
# 4. Re-encrypt all secrets with new key
kubectl get secrets --all-namespaces -o json | \
  kubectl replace -f -

# 5. Remove old key from EncryptionConfiguration
# 6. Restart API server again

When NOT to Use

• You don't have explicit written authorization to test
• Task is about defense/detection, not offense (use detection skills)
• You need to implement security controls (use implementing-* skills)
• Task requires compliance auditing (use auditing-* skills)
• You're investigating an incident (use incident response skills)
• Target is out of scope for your engagement
• Task is about vulnerability scanning only (use scanning tools)

Red Flags

• Performing actions without explicit written authorization from the asset owner
• Testing against production systems without a defined scope and rules of engagement
• Modifying cloud IAM policies or security groups without approval
• Exposing cloud credentials or secrets in logs or reports
• Running scans that generate excessive API calls and trigger billing alerts

Verification

• All steps executed successfully against a test environment before production use
• Output documented with screenshots or logs demonstrating expected behavior
• Cloud resource changes reverted or documented as intentional
• IAM policies reviewed for least-privilege compliance after testing
• No residual test resources left running (cost and security check)

References

• Kubernetes etcd Encryption Documentation
• CIS Kubernetes Benchmark - etcd Controls
• Securing etcd - K8s Security Guide
• Infosec: Encryption and etcd

Process

1. Analyze the task requirements
2. Apply domain expertise
3. Verify output quality

Anti-Rationalization Table