# Key Management Reference

> Comprehensive technical reference for cryptographic key management, including key lifecycle, KMS platforms, HSM integration, and compliance standards.

- Skill: `tools-only/key-management-reference` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add tools-only/key-management-reference`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tools-only/key-management-reference/raw
- Safety review: pending (external: skill-scanner WARNING, skillspector CAUTION)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: tools-only (https://skillmd.com/u/tools-only)
- Updated: 2026-09-29
- Page: https://skillmd.com/skills/tools-only/key-management-reference

---

# Key Management Reference

Comprehensive technical reference for cryptographic key management, including key lifecycle, KMS platforms, HSM integration, and compliance standards.

## Table of Contents

1. [Key Management Fundamentals](#key-management-fundamentals)
2. [Key Lifecycle](#key-lifecycle)
3. [Key Hierarchies](#key-hierarchies)
4. [Key Generation](#key-generation)
5. [Key Distribution](#key-distribution)
6. [Key Storage](#key-storage)
7. [Key Rotation](#key-rotation)
8. [Key Destruction](#key-destruction)
9. [KMS Platforms](#kms-platforms)
10. [HSM Integration](#hsm-integration)
11. [Secrets Management](#secrets-management)
12. [Access Control](#access-control)
13. [Compliance Standards](#compliance-standards)
14. [Key Escrow and Recovery](#key-escrow-and-recovery)
15. [Multi-Party Computation](#multi-party-computation)
16. [Audit and Monitoring](#audit-and-monitoring)
17. [Best Practices](#best-practices)
18. [Anti-Patterns](#anti-patterns)
19. [Implementation Examples](#implementation-examples)

---

## Key Management Fundamentals

### What is Key Management?

**Key Management** is the set of processes and procedures for generating, distributing, storing, rotating, and destroying cryptographic keys throughout their lifecycle.

**Core Principle**: The security of encrypted data depends entirely on the security of the encryption keys. Strong encryption with weak key management = insecure system.

### Why Key Management Matters

**Statistics**:
- 85% of data breaches involve compromised credentials or keys
- Average cost of a data breach: $4.45 million (IBM, 2023)
- Proper key management can prevent 90% of encryption-related breaches

**Key Management Risks**:

| Risk | Impact | Mitigation |
|------|--------|------------|
| **Key exposure** | All encrypted data compromised | HSM/KMS, access controls |
| **Key loss** | Data permanently inaccessible | Backup/escrow, redundancy |
| **Key theft** | Unauthorized decryption | Split knowledge, dual control |
| **Weak keys** | Cryptographic attacks | CSRNG, sufficient key length |
| **No rotation** | Large blast radius | Automated rotation policies |
| **Poor access control** | Insider threats | RBAC, audit logging |

### Key Management vs. Secret Management

| Aspect | Key Management | Secret Management |
|--------|---------------|-------------------|
| **Purpose** | Cryptographic operations | Application credentials |
| **Examples** | AES keys, RSA keys | API tokens, passwords, DB credentials |
| **Lifecycle** | Generate → Rotate → Destroy | Generate → Rotate → Revoke |
| **Storage** | HSM, KMS | Secret managers (Vault, AWS Secrets Manager) |
| **Access** | Programmatic (encrypt/decrypt) | Read/write credentials |
| **Rotation** | Months/years | Days/weeks |

**Note**: Modern platforms (e.g., HashiCorp Vault) handle both key management and secret management.

---

## Key Lifecycle

The key lifecycle consists of five primary phases:

```
┌──────────────┐
│ 1. Generation│
└──────┬───────┘
       │
       ▼
┌──────────────┐
│ 2. Distribution│
└──────┬───────┘
       │
       ▼
┌──────────────┐
│  3. Storage  │
└──────┬───────┘
       │
       ▼
┌──────────────┐
│ 4. Rotation  │
└──────┬───────┘
       │
       ▼
┌──────────────┐
│ 5. Destruction│
└──────────────┘
```

### Phase 1: Generation

**Objective**: Create cryptographically strong keys

**Requirements**:
- Use cryptographically secure random number generators (CSRNGs)
- Sufficient key length (AES-256, RSA-2048+, ECC-256+)
- Generate in secure environment (HSM, KMS, or secure enclave)
- Never derive keys from predictable sources

**Key Generation Sources**:

| Method | Security | Use Case |
|--------|----------|----------|
| **Hardware RNG** (HRNG) | Highest | HSM, TPM, secure enclaves |
| **OS CSRNG** (/dev/urandom, CryptGenRandom) | High | General application use |
| **KMS Generate** (AWS KMS, GCP KMS) | High | Cloud-native applications |
| **Deterministic RNG** (DRBG, CSPRNG) | Moderate | Embedded systems |
| **Weak RNG** (Math.random, timestamp) | ❌ Never use | N/A |

### Phase 2: Distribution

**Objective**: Securely transfer keys to authorized parties

**Methods**:

1. **Out-of-Band (OOB) Distribution**
   - Physical delivery (HSM devices, USB drives)
   - Separate communication channel (phone, SMS, email)
   - Use case: Initial master key setup

2. **Key Wrapping**
   - Encrypt data key with key encryption key (KEK)
   - Transport encrypted key over untrusted channel
   - Use case: Envelope encryption, cloud KMS

3. **Key Derivation**
   - Derive session keys from master key using KDF
   - No key transport needed
   - Use case: TLS session keys, password-based encryption

4. **Public Key Cryptography**
   - Encrypt symmetric key with recipient's public key
   - Use case: PGP, secure email, key exchange

**Distribution Security**:
```
Plaintext Key → ❌ NEVER transport unencrypted
Encrypted Key → ✅ Wrap with KEK
Key Reference → ✅ Use KMS key ID (no key material)
Derived Key   → ✅ Derive locally from shared secret
```

### Phase 3: Storage

**Objective**: Protect keys at rest

**Storage Tiers**:

| Tier | Security | Cost | Use Case |
|------|----------|------|----------|
| **Tier 1: HSM** | Highest | $$$$ | Root keys, CA keys, master keys |
| **Tier 2: KMS** | High | $$ | Application keys, data encryption keys |
| **Tier 3: Encrypted DB** | Moderate | $ | Encrypted DEKs, wrapped keys |
| **Tier 4: Keystore** | Moderate | $ | Local development, testing |
| **Tier 5: Plaintext** | ❌ Never | N/A | N/A |

### Phase 4: Rotation

**Objective**: Limit blast radius of key compromise

**Rotation Frequency**:
- **Root/Master keys**: Annually
- **Key Encryption Keys (KEKs)**: Quarterly
- **Data Encryption Keys (DEKs)**: Monthly or per dataset
- **Session keys**: Per session (ephemeral)
- **API keys**: 90 days
- **Emergency rotation**: Immediately upon suspected compromise

### Phase 5: Destruction

**Objective**: Securely delete keys when no longer needed

**Destruction Methods**:

| Method | Security | Use Case |
|--------|----------|----------|
| **Cryptographic erasure** | Highest | Destroy KEK (makes DEKs unrecoverable) |
| **Secure overwrite** | High | Multi-pass overwrite (DOD 5220.22-M) |
| **HSM destroy** | High | Hardware-level key zeroization |
| **Shredding** | Moderate | Physical destruction of storage media |
| **Delete file** | ❌ Insufficient | Data may be recoverable |

**Key Destruction Timeline**:
1. **Deprecation**: Mark key as deprecated (no new encryptions)
2. **Grace period**: 30-90 days (allow existing decryptions)
3. **Scheduled deletion**: Schedule destruction with waiting period
4. **Audit log**: Record destruction event
5. **Verification**: Confirm key is irrecoverable

---

## Key Hierarchies

### Concept

**Key Hierarchy**: Organize keys in layers where upper-level keys encrypt lower-level keys.

**Benefits**:
- Efficient key rotation (rotate KEK without re-encrypting data)
- Separation of concerns (different keys for different purposes)
- Reduced attack surface (root keys rarely used)
- Granular access control (different teams manage different key levels)

### Three-Tier Hierarchy

```
┌─────────────────────────────────────────────┐
│ Level 1: Root Key (Master Key)              │  ← Stored in HSM
│ - Never rotates (or rotates rarely)         │  ← Rarely used
│ - Used to encrypt KEKs                      │  ← Highest security
│ - FIPS 140-2 Level 3+ HSM                   │
└───────────────────┬─────────────────────────┘
                    │ Encrypts
                    ▼
┌─────────────────────────────────────────────┐
│ Level 2: Key Encryption Key (KEK)           │  ← Stored in KMS or encrypted DB
│ - Rotates periodically (e.g., quarterly)    │  ← Used to encrypt DEKs
│ - Used to encrypt DEKs                      │  ← Medium security
│ - One KEK per application/tenant            │
└───────────────────┬─────────────────────────┘
                    │ Encrypts
                    ▼
┌─────────────────────────────────────────────┐
│ Level 3: Data Encryption Key (DEK)          │  ← Stored encrypted with data
│ - Rotates frequently (e.g., per file/record)│  ← Used to encrypt actual data
│ - Used to encrypt data                      │  ← Performance-critical
│ - One DEK per file/record/tenant            │
└───────────────────┬─────────────────────────┘
                    │ Encrypts
                    ▼
            ┌──────────────────┐
            │  Encrypted Data  │  ← Application data
            └──────────────────┘
```

### Key Hierarchy Example

**Scenario**: Encrypt user files in cloud storage

```python
# Level 1: Root Key (stored in HSM)
root_key = hsm.get_key("root-key-2024")

# Level 2: KEK for this application (encrypted by root key)
kek_plaintext = generate_random_key(32)
kek_encrypted = hsm.encrypt(root_key, kek_plaintext)
kms.store("app-kek-2024-Q4", kek_encrypted)

# Level 3: DEK for each file (encrypted by KEK)
def encrypt_file(file_data):
    # Generate DEK
    dek = generate_random_key(32)

    # Encrypt file with DEK
    ciphertext = aes_gcm_encrypt(dek, file_data)

    # Encrypt DEK with KEK
    kek = kms.decrypt("app-kek-2024-Q4", root_key)
    encrypted_dek = aes_gcm_encrypt(kek, dek)

    # Store encrypted DEK + ciphertext
    return {
        "encrypted_dek": encrypted_dek,
        "ciphertext": ciphertext
    }
```

### Key Rotation with Hierarchies

**Advantage**: Rotate KEK without re-encrypting data

**Process**:
1. Generate new KEK
2. Decrypt DEKs with old KEK
3. Re-encrypt DEKs with new KEK
4. Data remains encrypted with DEKs (no change)

**Example**:
```python
def rotate_kek(old_kek, new_kek):
    """Rotate KEK without touching data"""
    records = db.query("SELECT id, encrypted_dek FROM files")

    for record in records:
        # Decrypt DEK with old KEK
        dek = decrypt(old_kek, record.encrypted_dek)

        # Re-encrypt DEK with new KEK
        new_encrypted_dek = encrypt(new_kek, dek)

        # Update record (data ciphertext unchanged)
        db.update("files", record.id, {
            "encrypted_dek": new_encrypted_dek
        })
```

**Performance**: Re-encrypting DEKs (small) is 1000x faster than re-encrypting data (large).

---

## Key Generation

### Cryptographically Secure Random Number Generators (CSRNGs)

**Requirements for Cryptographic Keys**:
1. **Unpredictability**: Cannot predict future or past outputs
2. **Uniformity**: All values equally likely
3. **Sufficient entropy**: At least 256 bits for AES-256
4. **Non-deterministic**: Not based on predictable seeds

### Platform-Specific CSRNGs

#### Linux: /dev/urandom

**Source**: Kernel CSPRNG (Yarrow, Fortuna algorithms)

**Entropy Sources**:
- Hardware RNG (if available: RDRAND, RDSEED)
- Interrupt timing
- Disk I/O timing
- Network packet timing

**Usage**:
```bash
# Generate 32-byte key (256 bits)
dd if=/dev/urandom of=key.bin bs=32 count=1

# Generate hex-encoded key
hexdump -n 32 -e '32/1 "%02x" "\n"' /dev/urandom
```

**Python**:
```python
import os

# Generate 256-bit key
key = os.urandom(32)  # 32 bytes = 256 bits

# Or use secrets module (Python 3.6+)
import secrets
key = secrets.token_bytes(32)
hex_key = secrets.token_hex(32)  # 64 hex characters
```

**Note**: `/dev/random` vs `/dev/urandom`
- `/dev/random`: Blocks when entropy low (not needed for modern kernels)
- `/dev/urandom`: Never blocks (preferred for key generation)

#### Windows: CryptGenRandom

**Source**: Windows CryptoAPI CSPRNG

**Python**:
```python
import os

# os.urandom() uses CryptGenRandom on Windows
key = os.urandom(32)
```

**C#**:
```csharp
using System.Security.Cryptography;

// Generate 256-bit key
byte[] key = new byte[32];
using (var rng = RandomNumberGenerator.Create())
{
    rng.GetBytes(key);
}
```

#### macOS: /dev/random

**Source**: Kernel CSPRNG (Yarrow algorithm)

**Usage**: Same as Linux /dev/urandom

```python
import os
key = os.urandom(32)
```

### Hardware RNGs

#### Intel RDRAND/RDSEED

**What**: CPU instruction for hardware random number generation

**Check Support**:
```bash
grep -o rdrand /proc/cpuinfo
grep -o rdseed /proc/cpuinfo
```

**C Example**:
```c
#include <immintrin.h>
#include <stdint.h>

uint64_t get_random_uint64() {
    uint64_t rand;
    while (!_rdrand64_step(&rand)) {
        // Retry if RDRAND fails
    }
    return rand;
}
```

**Note**: Linux /dev/urandom automatically uses RDRAND if available.

#### TPM (Trusted Platform Module)

**What**: Hardware chip for cryptographic operations and key storage

**Generate Key**:
```bash
# Using tpm2-tools
tpm2_getrandom 32 > key.bin
```

**Python (tpm2-pytss)**:
```python
from tpm2_pytss import ESAPI

esapi = ESAPI()
random_bytes = esapi.get_random(32)
```

### Key Derivation Functions (KDFs)

**Use Case**: Derive cryptographic keys from passwords or shared secrets

**Algorithms**:

| Algorithm | Standard | Security | Use Case |
|-----------|----------|----------|----------|
| **Argon2id** | RFC 9106 | Highest | Password hashing, key derivation |
| **scrypt** | RFC 7914 | High | Key derivation (memory-hard) |
| **PBKDF2** | RFC 8018 | Moderate | FIPS compliance, legacy systems |
| **HKDF** | RFC 5869 | High | Key derivation from shared secrets |
| **bcrypt** | - | Moderate | Password hashing only |

#### Argon2id (Recommended)

**Parameters**:
- `time_cost`: Number of iterations (min 3)
- `memory_cost`: Memory in KB (min 64 MB)
- `parallelism`: Number of threads (min 4)
- `salt`: Random salt (min 16 bytes)

**Python Example**:
```python
from argon2 import PasswordHasher
from argon2.low_level import hash_secret_raw, Type
import secrets

password = b"user-password"
salt = secrets.token_bytes(16)

# Derive 256-bit key
key = hash_secret_raw(
    secret=password,
    salt=salt,
    time_cost=3,          # Iterations
    memory_cost=65536,    # 64 MB
    parallelism=4,        # 4 threads
    hash_len=32,          # 256-bit key
    type=Type.ID          # Argon2id
)
```

**Recommended Parameters** (2024):
- Time cost: 3-4
- Memory cost: 65536-131072 (64-128 MB)
- Parallelism: 4
- Hash length: 32 bytes (256 bits)

#### HKDF (HMAC-based KDF)

**Use Case**: Derive multiple keys from single master key

**Python Example**:
```python
from cryptography.hazmat.primitives.kdf.hkdf import HKDF
from cryptography.hazmat.primitives import hashes
import secrets

# Master key material
ikm = secrets.token_bytes(32)  # Input key material

# Derive multiple keys
hkdf = HKDF(
    algorithm=hashes.SHA256(),
    length=32,
    salt=None,
    info=b"encryption-key"
)
encryption_key = hkdf.derive(ikm)

hkdf = HKDF(
    algorithm=hashes.SHA256(),
    length=32,
    salt=None,
    info=b"authentication-key"
)
auth_key = hkdf.derive(ikm)
```

### Key Length Requirements

**Symmetric Encryption**:
- **AES**: 128, 192, or 256 bits (use 256)
- **ChaCha20**: 256 bits

**Asymmetric Encryption**:
- **RSA**: Minimum 2048 bits (3072+ recommended)
- **ECC**: Minimum 256 bits (384+ for high security)
- **Ed25519**: 256 bits (fixed)

**Key Length Security Equivalence**:

| Symmetric | RSA | ECC | Security Level |
|-----------|-----|-----|----------------|
| 128 bits | 3072 bits | 256 bits | Medium (until 2030) |
| 192 bits | 7680 bits | 384 bits | High |
| 256 bits | 15360 bits | 512 bits | Very High |

---

## Key Distribution

### Out-of-Band (OOB) Distribution

**Method**: Deliver keys through separate, secure channel

**Use Cases**:
- Initial master key setup
- Root CA key distribution
- HSM key loading
- Emergency key recovery

**Examples**:

1. **Physical Delivery**
   - USB drive with encrypted key
   - Smart card
   - HSM device
   - Sealed envelope

2. **Separate Communication Channel**
   - Phone call
   - SMS (for low-security scenarios)
   - Separate email system
   - In-person transfer

**Best Practices**:
- Split key into multiple parts (M-of-N scheme)
- Require multiple people (dual control)
- Audit trail for key delivery
- Time-limited validity

### Key Wrapping (Envelope Encryption)

**Concept**: Encrypt data key with key encryption key (KEK) for secure transport

**Process**:
```
Sender                        Transport Channel                    Receiver
──────                        ─────────────────                    ────────
Generate DEK
    │
    ▼
Encrypt Data
with DEK
    │
    ▼
Wrap DEK
with KEK
    │
    ▼
Send Wrapped DEK  ─────────────────────────────────────>  Receive
+ Ciphertext                                                       │
                                                                   ▼
                                                              Unwrap DEK
                                                              with KEK
                                                                   │
                                                                   ▼
                                                              Decrypt Data
                                                              with DEK
```

**Key Wrapping Algorithms**:

| Algorithm | Standard | Key Sizes | Use Case |
|-----------|----------|-----------|----------|
| **AES-KW** | RFC 3394 | 128, 192, 256 | Symmetric key wrapping |
| **AES-KWP** | RFC 5649 | 128, 192, 256 | Padded key wrapping |
| **RSA-OAEP** | PKCS#1 v2.1 | 2048+ | Asymmetric key wrapping |
| **AES-GCM** | NIST SP 800-38D | 128, 192, 256 | Authenticated key wrapping |

**Python Example (AES Key Wrap)**:
```python
from cryptography.hazmat.primitives.keywrap import aes_key_wrap, aes_key_unwrap
import secrets

# KEK (256-bit)
kek = secrets.token_bytes(32)

# DEK to wrap (256-bit)
dek = secrets.token_bytes(32)

# Wrap DEK with KEK
wrapped_dek = aes_key_wrap(kek, dek, backend=default_backend())

# Transport wrapped_dek over untrusted channel

# Unwrap DEK
unwrapped_dek = aes_key_unwrap(kek, wrapped_dek, backend=default_backend())

assert unwrapped_dek == dek
```

### Key Agreement Protocols

**Use Case**: Two parties establish shared secret without transmitting keys

#### Diffie-Hellman (DH)

**Classic DH**:
```
Alice                                    Bob
─────                                    ───
Generate private key a                   Generate private key b
Compute public key A = g^a mod p         Compute public key B = g^b mod p
                A ──────────────────→
                ←────────────────── B
Compute shared secret:                   Compute shared secret:
s = B^a mod p                            s = A^b mod p

Both derive encryption key from s using KDF
```

**Python Example (ECDH - Elliptic Curve DH)**:
```python
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives.kdf.hkdf import HKDF
from cryptography.hazmat.primitives import hashes

# Alice generates key pair
alice_private = ec.generate_private_key(ec.SECP256R1())
alice_public = alice_private.public_key()

# Bob generates key pair
bob_private = ec.generate_private_key(ec.SECP256R1())
bob_public = bob_private.public_key()

# Alice computes shared secret
alice_shared_key = alice_private.exchange(ec.ECDH(), bob_public)

# Bob computes shared secret
bob_shared_key = bob_private.exchange(ec.ECDH(), alice_public)

# Both derive encryption key from shared secret
def derive_key(shared_secret):
    return HKDF(
        algorithm=hashes.SHA256(),
        length=32,
        salt=None,
        info=b"handshake"
    ).derive(shared_secret)

alice_key = derive_key(alice_shared_key)
bob_key = derive_key(bob_shared_key)

assert alice_key == bob_key  # Same encryption key
```

### Key Distribution via KMS

**Concept**: Use cloud KMS to manage key distribution

**AWS KMS Example**:
```python
import boto3

kms = boto3.client('kms')

# Sender: Generate data key
response = kms.generate_data_key(
    KeyId='arn:aws:kms:us-east-1:123456789012:key/12345678',
    KeySpec='AES_256'
)

dek_plaintext = response['Plaintext']      # Use to encrypt data
dek_encrypted = response['CiphertextBlob']  # Distribute this

# Encrypt data with DEK
ciphertext = encrypt_data(plaintext, dek_plaintext)

# Send encrypted DEK + ciphertext to receiver

# Receiver: Decrypt DEK
response = kms.decrypt(CiphertextBlob=dek_encrypted)
dek = response['Plaintext']

# Decrypt data
plaintext = decrypt_data(ciphertext, dek)
```

---

## Key Storage

### Storage Security Requirements

**NIST SP 800-57 Requirements**:
1. **Confidentiality**: Keys must be encrypted or in tamper-resistant hardware
2. **Integrity**: Detect unauthorized modification
3. **Availability**: Keys accessible when needed (but not more)
4. **Accountability**: Audit who accessed keys
5. **Authenticity**: Verify key source

### Storage Tiers

#### Tier 1: Hardware Security Module (HSM)

**What**: Dedicated hardware device for cryptographic operations and key storage

**Features**:
- Keys never leave HSM (operations performed inside)
- FIPS 140-2 Level 3 or 4 certified
- Tamper-resistant/tamper-evident
- Physical security controls
- Audit logging
- High-availability clustering

**FIPS 140-2 Levels**:

| Level | Requirements | Use Case |
|-------|-------------|----------|
| **Level 1** | Basic security | Software implementations |
| **Level 2** | Tamper-evident seals, role-based auth | General enterprise |
| **Level 3** | Tamper-resistant hardware, identity-based auth | Financial, healthcare |
| **Level 4** | Environmental protection (voltage, temp) | Government, military |

**HSM Vendors**:
- Thales Luna HSM
- nCipher nShield
- Utimaco SecurityServer
- AWS CloudHSM
- Google Cloud HSM
- Azure Dedicated HSM

**AWS CloudHSM Example**:
```python
import boto3

cloudhsm = boto3.client('cloudhsmv2')

# Create HSM cluster
response = cloudhsm.create_cluster(
    SubnetIds=['subnet-12345', 'subnet-67890'],
    HsmType='hsm1.medium'
)

cluster_id = response['Cluster']['ClusterId']

# Initialize cluster and create HSM
# Then use PKCS#11 or JCE provider to interact with HSM
```

**Cost**: $10,000-$100,000+ for hardware HSM, $1-5/hour for cloud HSM

#### Tier 2: Cloud Key Management Service (KMS)

**What**: Managed service for key storage and encryption operations

**Features**:
- API-based key management
- Automatic key rotation
- Access control via IAM
- Audit logging
- Regional replication
- Cost-effective ($1/month per key)

**AWS KMS**:
```python
import boto3

kms = boto3.client('kms')

# Create key
response = kms.create_key(
    Description='Application encryption key',
    KeyUsage='ENCRYPT_DECRYPT',
    Origin='AWS_KMS'
)

key_id = response['KeyMetadata']['KeyId']

# Create alias
kms.create_alias(
    AliasName='alias/app-key',
    TargetKeyId=key_id
)

# Enable automatic rotation
kms.enable_key_rotation(KeyId=key_id)

# Encrypt data
ciphertext = kms.encrypt(
    KeyId='alias/app-key',
    Plaintext=b'Secret data'
)['CiphertextBlob']

# Decrypt data
plaintext = kms.decrypt(CiphertextBlob=ciphertext)['Plaintext']
```

**Google Cloud KMS**:
```python
from google.cloud import kms

client = kms.KeyManagementServiceClient()

# Create key ring
key_ring_path = f'projects/{project_id}/locations/{location}/keyRings/{key_ring_id}'
client.create_key_ring(request={'parent': location_path, 'key_ring_id': key_ring_id})

# Create crypto key
key_path = f'{key_ring_path}/cryptoKeys/{key_id}'
client.create_crypto_key(
    request={
        'parent': key_ring_path,
        'crypto_key_id': key_id,
        'crypto_key': {'purpose': kms.CryptoKey.CryptoKeyPurpose.ENCRYPT_DECRYPT}
    }
)

# Encrypt
encrypt_response = client.encrypt(
    request={'name': key_path, 'plaintext': b'Secret data'}
)
ciphertext = encrypt_response.ciphertext

# Decrypt
decrypt_response = client.decrypt(
    request={'name': key_path, 'ciphertext': ciphertext}
)
plaintext = decrypt_response.plaintext
```

**Azure Key Vault**:
```python
from azure.identity import DefaultAzureCredential
from azure.keyvault.keys import KeyClient
from azure.keyvault.keys.crypto import CryptographyClient, EncryptionAlgorithm

credential = DefaultAzureCredential()
key_client = KeyClient(vault_url=f'https://{vault_name}.vault.azure.net/', credential=credential)

# Create key
key = key_client.create_rsa_key('app-key', size=2048)

# Encrypt
crypto_client = CryptographyClient(key, credential)
result = crypto_client.encrypt(EncryptionAlgorithm.rsa_oaep, b'Secret data')
ciphertext = result.ciphertext

# Decrypt
result = crypto_client.decrypt(EncryptionAlgorithm.rsa_oaep, ciphertext)
plaintext = result.plaintext
```

#### Tier 3: Encrypted Database

**What**: Store encrypted keys in database (KEK stored separately)

**Schema Example**:
```sql
CREATE TABLE encryption_keys (
    id SERIAL PRIMARY KEY,
    key_id VARCHAR(255) UNIQUE NOT NULL,
    encrypted_key BYTEA NOT NULL,      -- DEK encrypted by KEK
    key_nonce BYTEA NOT NULL,          -- Nonce for encryption
    key_type VARCHAR(50) NOT NULL,     -- "AES-256", "RSA-2048", etc.
    created_at TIMESTAMP NOT NULL,
    rotated_at TIMESTAMP,
    expires_at TIMESTAMP,
    status VARCHAR(20) NOT NULL        -- "active", "deprecated", "destroyed"
);

CREATE INDEX idx_key_id ON encryption_keys(key_id);
CREATE INDEX idx_status ON encryption_keys(status);
```

**Application Example**:
```python
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
import secrets

class KeyStore:
    def __init__(self, kek):
        """KEK (Key Encryption Key) for encrypting DEKs"""
        self.kek = kek
        self.aesgcm = AESGCM(kek)

    def store_key(self, key_id, dek, key_type):
        """Store encrypted DEK in database"""
        nonce = secrets.token_bytes(12)
        encrypted_key = self.aesgcm.encrypt(nonce, dek, None)

        db.execute("""
            INSERT INTO encryption_keys (key_id, encrypted_key, key_nonce, key_type, status, created_at)
            VALUES (?, ?, ?, ?, 'active', NOW())
        """, (key_id, encrypted_key, nonce, key_type))

    def retrieve_key(self, key_id):
        """Retrieve and decrypt DEK"""
        row = db.query("""
            SELECT encrypted_key, key_nonce
            FROM encryption_keys
            WHERE key_id = ? AND status = 'active'
        """, (key_id,))

        if not row:
            raise ValueError(f"Key not found: {key_id}")

        dek = self.aesgcm.decrypt(row['key_nonce'], row['encrypted_key'], None)
        return dek
```

#### Tier 4: Operating System Keystore

**What**: OS-provided secure storage for keys and credentials

**macOS Keychain**:
```python
import keyring

# Store key
keyring.set_password('myapp', 'encryption_key', base64_encoded_key)

# Retrieve key
key = keyring.get_password('myapp', 'encryption_key')
```

**Windows Credential Manager**:
```python
import keyring

# Same API as macOS
keyring.set_password('myapp', 'encryption_key', base64_encoded_key)
key = keyring.get_password('myapp', 'encryption_key')
```

**Linux Secret Service** (GNOME Keyring, KWallet):
```python
import secretstorage

connection = secretstorage.dbus_init()
collection = secretstorage.get_default_collection(connection)

# Store key
collection.create_item(
    'myapp-encryption-key',
    {'application': 'myapp'},
    base64_encoded_key
)

# Retrieve key
items = collection.search_items({'application': 'myapp'})
key = items[0].get_secret()
```

### Key Storage Best Practices

1. **Never store keys in plaintext**
   ```
   ❌ config.yaml: encryption_key: "abc123..."
   ✅ Use KMS, HSM, or encrypted storage
   ```

2. **Separate keys from data**
   ```
   ❌ Same database/server
   ✅ Separate KMS service or HSM
   ```

3. **Use key hierarchies**
   ```
   Root Key (HSM) → KEK (KMS) → DEK (Encrypted DB) → Data
   ```

4. **Access control**
   ```
   ✅ Role-based access (RBAC)
   ✅ Principle of least privilege
   ✅ Audit logging
   ```

5. **Key backup and redundancy**
   ```
   ✅ Multi-region KMS replication
   ✅ HSM cluster (HA)
   ✅ Offline backup (encrypted, offline storage)
   ```

6. **Separate keys by environment**
   ```
   dev-encryption-key
   staging-encryption-key
   prod-encryption-key
   ```

---

## Key Rotation

### Why Rotate Keys?

**Reasons**:
1. **Limit blast radius**: Compromised old key doesn't affect new data
2. **Compliance**: Many standards require periodic rotation (PCI-DSS, HIPAA)
3. **Cryptographic hygiene**: Reduce ciphertext under single key
4. **Mitigate key exposure**: Employee departure, system compromise, vulnerability disclosure

**Rotation Frequency Recommendations**:

| Key Type | Frequency | Rationale |
|----------|-----------|-----------|
| **Root/Master keys** | Annually | Rarely used, highest security |
| **KEKs** | Quarterly | Balance security and operational overhead |
| **DEKs** | Monthly or per-dataset | Frequent rotation, minimal overhead with envelope encryption |
| **Session keys** | Per-session | Ephemeral, perfect forward secrecy |
| **API keys** | 90 days | Compliance requirement (SOC 2, PCI-DSS) |
| **TLS certificates** | 90 days (Let's Encrypt) or annually | Industry standard |
| **Database passwords** | 90 days | Compliance requirement |
| **Emergency** | Immediately | Suspected compromise |

### Rotation Strategies

#### Strategy 1: Re-encrypt All Data (Complete Rotation)

**Process**:
1. Generate new key
2. Decrypt all data with old key
3. Encrypt all data with new key
4. Update key references
5. Delete old key (after grace period)

**Pros**:
- Simple conceptually
- Complete forward secrecy

**Cons**:
- Expensive (re-encrypt all data)
- Downtime required
- Risk of data loss during migration

**Use Case**: Small datasets, infrequent rotation

**Example**:
```python
def rotate_complete(old_key, new_key):
    """Complete key rotation - re-encrypt all data"""
    records = db.query("SELECT id, ciphertext FROM encrypted_data")

    for record in records:
        # Decrypt with old key
        plaintext = decrypt(old_key, record.ciphertext)

        # Re-encrypt with new key
        new_ciphertext = encrypt(new_key, plaintext)

        # Update record
        db.update("encrypted_data", record.id, {"ciphertext": new_ciphertext})

    # Mark old key as deprecated
    mark_key_deprecated(old_key)
```

#### Strategy 2: Envelope Encryption (Zero-Downtime Rotation)

**Process**:
1. Generate new KEK
2. Re-encrypt DEKs with new KEK
3. Data stays encrypted with DEKs (no change)
4. Update KEK references
5. Delete old KEK

**Pros**:
- Fast (only re-encrypt small DEKs, not large data)
- No downtime
- Scales to large datasets

**Cons**:
- Requires key hierarchy

**Use Case**: Large datasets, cloud KMS, production systems

**Example**:
```python
def rotate_envelope(old_kek, new_kek):
    """Envelope encryption rotation - only re-encrypt DEKs"""
    records = db.query("SELECT id, encrypted_dek, dek_nonce FROM files")

    old_aesgcm = AESGCM(old_kek)
    new_aesgcm = AESGCM(new_kek)

    for record in records:
        # Decrypt DEK with old KEK
        dek = old_aesgcm.decrypt(record.dek_nonce, record.encrypted_dek, None)

        # Re-encrypt DEK with new KEK
        new_nonce = secrets.token_bytes(12)
        new_encrypted_dek = new_aesgcm.encrypt(new_nonce, dek, None)

        # Update encrypted DEK (data ciphertext unchanged)
        db.update("files", record.id, {
            "encrypted_dek": new_encrypted_dek,
            "dek_nonce": new_nonce
        })

    # Mark old KEK as deprecated
    mark_key_deprecated(old_kek)
```

**Performance Comparison**:
```
Dataset: 1 TB encrypted data
Complete Rotation: 1 TB re-encryption = hours/days
Envelope Rotation: 10,000 DEKs * 32 bytes = 320 KB re-encryption = seconds
```

#### Strategy 3: Versioned Keys (Multi-Version)

**Process**:
1. Keep both old and new keys active
2. Write new data with new key
3. Read old data with old key
4. Gradually re-encrypt old data (background job)
5. Retire old key after migration complete

**Pros**:
- Zero downtime
- Gradual migration
- No service interruption

**Cons**:
- Complex (track key versions)
- Multiple keys active simultaneously

**Use Case**: Large datasets, continuous writes, cannot afford downtime

**Schema**:
```sql
CREATE TABLE encrypted_data (
    id SERIAL PRIMARY KEY,
    ciphertext BYTEA NOT NULL,
    key_version INTEGER NOT NULL,  -- Track which key encrypted this row
    nonce BYTEA NOT NULL,
    created_at TIMESTAMP NOT NULL,
    updated_at TIMESTAMP NOT NULL
);

CREATE INDEX idx_key_version ON encrypted_data(key_version);
```

**Application**:
```python
class VersionedKeyManager:
    def __init__(self):
        self.keys = {
            1: load_key("key-v1"),  # Old key
            2: load_key("key-v2"),  # New key (active)
        }
        self.active_version = 2

    def encrypt(self, plaintext):
        """Encrypt with active key version"""
        key = self.keys[self.active_version]
        nonce = secrets.token_bytes(12)
        ciphertext = AESGCM(key).encrypt(nonce, plaintext, None)

        return {
            "ciphertext": ciphertext,
            "key_version": self.active_version,
            "nonce": nonce
        }

    def decrypt(self, ciphertext, key_version, nonce):
        """Decrypt with appropriate key version"""
        if key_version not in self.keys:
            raise ValueError(f"Unknown key version: {key_version}")

        key = self.keys[key_version]
        plaintext = AESGCM(key).decrypt(nonce, ciphertext, None)
        return plaintext

    def background_reencryption(self, batch_size=100):
        """Gradually re-encrypt old data"""
        while True:
            # Get batch of old records
            records = db.query("""
                SELECT id, ciphertext, key_version, nonce
                FROM encrypted_data
                WHERE key_version < ?
                LIMIT ?
            """, (self.active_version, batch_size))

            if not records:
                break  # Migration complete

            for record in records:
                # Decrypt with old key
                plaintext = self.decrypt(
                    record.ciphertext,
                    record.key_version,
                    record.nonce
                )

                # Re-encrypt with new key
                encrypted = self.encrypt(plaintext)

                # Update record
                db.update("encrypted_data", record.id, encrypted)

            time.sleep(1)  # Rate limit
```

### AWS KMS Automatic Key Rotation

**How It Works**:
- AWS rotates key material annually (automatic)
- Old key versions retained for decryption
- New encryptions use new key version
- Application code unchanged (transparent)

**Enable Automatic Rotation**:
```bash
aws kms enable-key-rotation --key-id 12345678-1234-1234-1234-123456789012
```

**Check Rotation Status**:
```bash
aws kms get-key-rotation-status --key-id 12345678-1234-1234-1234-123456789012
```

**How Versioning Works Internally**:
```
Key ARN: arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012

Key Material Versions:
- v1 (2023-01-01): Used for decrypting old ciphertext
- v2 (2024-01-01): Used for decrypting old ciphertext
- v3 (2025-01-01): Current version (used for new encryptions)

When you call kms.encrypt():
  → Uses v3 (current key material)

When you call kms.decrypt(ciphertext):
  → KMS detects which version encrypted it
  → Uses correct key material automatically
  → Application doesn't need to track versions
```

### Manual Rotation Procedure

**Pre-Rotation Checklist**:
- [ ] Backup current keys (encrypted)
- [ ] Test decryption with old key
- [ ] Verify access to generate new keys
- [ ] Schedule maintenance window (if needed)
- [ ] Notify stakeholders
- [ ] Prepare rollback plan

**Rotation Steps**:

1. **Generate new key**
   ```python
   new_key = secrets.token_bytes(32)
   ```

2. **Store new key securely**
   ```python
   kms.create_key(Description='app-key-2025-Q1')
   ```

3. **Update application config** (versioned keys)
   ```yaml
   encryption_keys:
     - id: key-2024-Q4
       version: 1
       status: deprecated
     - id: key-2025-Q1
       version: 2
       status: active
   ```

4. **Re-encrypt data** (choose strategy: complete, envelope, or versioned)

5. **Verify rotation**
   ```python
   # Test encryption with new key
   ciphertext = encrypt(new_key, test_data)
   plaintext = decrypt(new_key, ciphertext)
   assert plaintext == test_data

   # Test decryption of old data
   old_plaintext = decrypt(old_key, old_ciphertext)
   assert old_plaintext == expected_data
   ```

6. **Monitor for errors**
   - Check application logs
   - Monitor decryption failures
   - Track key usage metrics

7. **Grace period** (30-90 days)
   - Keep old key active for decryption
   - Monitor usage of old key
   - Ensure all data migrated

8. **Delete old key**
   ```python
   kms.schedule_key_deletion(KeyId=old_key_id, PendingWindowInDays=30)
   ```

### Rotation Automation

**Automated Rotation Script**:
```python
#!/usr/bin/env python3
import boto3
import datetime

class KeyRotationAutomation:
    def __init__(self, kms_client):
        self.kms = kms_client

    def check_rotation_needed(self, key_id):
        """Check if key needs rotation based on age"""
        response = self.kms.describe_key(KeyId=key_id)
        creation_date = response['KeyMetadata']['CreationDate']
        age_days = (datetime.datetime.now(datetime.timezone.utc) - creation_date).days

        # Rotate if key is > 365 days old
        return age_days > 365

    def rotate_key(self, key_id):
        """Rotate KMS key"""
        # Enable automatic rotation
        self.kms.enable_key_rotation(KeyId=key_id)

        # Or manually create new key version
        # (for customer-managed keys)
        response = self.kms.create_key(
            Description=f'Rotated from {key_id}',
            Origin='AWS_KMS'
        )

        new_key_id = response['KeyMetadata']['KeyId']

        # Update alias to point to new key
        self.kms.update_alias(
            AliasName='alias/my-key',
            TargetKeyId=new_key_id
        )

        return new_key_id

    def scan_and_rotate(self):
        """Scan all keys and rotate if needed"""
        response = self.kms.list_keys()

        for key in response['Keys']:
            key_id = key['KeyId']

            if self.check_rotation_needed(key_id):
                print(f"Rotating key: {key_id}")
                new_key_id = self.rotate_key(key_id)
                print(f"New key: {new_key_id}")

# Run automated rotation
kms = boto3.client('kms')
automation = KeyRotationAutomation(kms)
automation.scan_and_rotate()
```

---

## Key Destruction

### Secure Key Deletion

**Objective**: Ensure keys cannot be recovered after deletion

### Deletion Methods

#### Method 1: Cryptographic Erasure

**Concept**: Destroy KEK, making all DEKs (and data) irrecoverable

**Advantages**:
- Instant (delete one KEK)
- Guaranteed irrecoverability (even with data backups)
- No need to overwrite data

**Process**:
```
Before:
Root Key → KEK → DEK1, DEK2, DEK3, ... → Encrypted Data

After Cryptographic Erasure (Delete KEK):
Root Key → ❌ KEK DELETED ❌
           ↓
       DEK1, DEK2, DEK3 (irrecoverable)
           ↓
       Encrypted Data (permanently inaccessible)
```

**Example**:
```python
def cryptographic_erasure(kek_id):
    """Delete KEK to make all DEKs irrecoverable"""
    # Schedule KEK deletion
    kms.schedule_key_deletion(
        KeyId=kek_id,
        PendingWindowInDays=30  # 7-30 day waiting period
    )

    # After waiting period, KEK is permanently deleted
    # All DEKs encrypted by this KEK become irrecoverable
    # Data encrypted by those DEKs is permanently inaccessible
```

**Use Case**: Delete all user data (GDPR "right to be forgotten")

#### Method 2: Secure Overwrite

**Concept**: Overwrite key data multiple times before deletion

**Standards**:
- **DOD 5220.22-M**: 3-pass overwrite (0xFF, 0x00, random)
- **Gutmann Method**: 35-pass overwrite (overkill for modern drives)
- **NIST SP 800-88**: 1-pass overwrite sufficient for modern drives

**Bash Example**:
```bash
# DOD 5220.22-M (3-pass)
shred -vfz -n 3 key.bin

# Gutmann method (35-pass)
shred -vfz -n 35 key.bin

# Single-pass (sufficient for SSDs)
shred -vfz -n 1 key.bin


…(truncated)