AWS CloudHSM Diagnostics
When to use
Any AWS CloudHSM investigation where the console alone is insufficient — cluster and HSM management, client connectivity, cryptographic key operations, HSM user management, backup and restore, high availability, SDK integration, PKCS#11 library, JCE provider, or OpenSSL engine issues.
Investigation workflow
Step 1 — Collect and triage
aws cloudhsmv2 describe-clusters
aws cloudhsmv2 describe-clusters --filters clusterIds=<cluster-id>
aws cloudhsmv2 list-tags --resource-id <cluster-id>
Step 2 — Domain deep dive
aws cloudhsmv2 describe-backups --filters clusterIds=<cluster-id>
aws ec2 describe-security-groups --group-ids <cluster-sg-id>
aws ec2 describe-subnets --subnet-ids <subnet-id>
# On client instance:
/opt/cloudhsm/bin/cloudhsm-cli cluster info
/opt/cloudhsm/bin/cloudhsm-cli user list
Step 3 — Detailed investigation
aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=cloudhsmv2.amazonaws.com --max-results 20
aws cloudwatch get-metric-statistics --namespace AWS/CloudHSM --metric-name HsmKeysSessionCount --dimensions Name=ClusterId,Value=<id> --start-time <start> --end-time <end> --period 300 --statistics Average
# Check client logs:
# /opt/cloudhsm/run/cloudhsm-client.log
Read references/guardrails.md before concluding on any CloudHSM issue.
Tool quick reference
| Tool / API |
When to use |
describe-clusters |
Check cluster and HSM status |
describe-backups |
Check backup status and history |
cloudhsm-cli |
HSM-level user and key management |
| Security Groups |
Verify client-to-HSM connectivity |
| CloudWatch Metrics |
Monitor HSM health and usage |
| CloudTrail |
Audit management API calls |
| Client logs |
Debug connectivity and operation errors |
Gotchas: AWS CloudHSM
- CloudHSM is a single-tenant HSM. You manage HSM users, keys, and access. AWS manages hardware and firmware. AWS cannot access your keys or recover them if credentials are lost. Losing all crypto officer (CO) credentials means losing key management access permanently.
- Client connectivity requires: (1) CloudHSM client installed on EC2 instance, (2) instance in same VPC or peered VPC, (3) security group allows port 2223-2225 from client to HSM ENI, (4) client configuration points to cluster IP.
- HSM users are managed INSIDE the HSM, not via IAM. Three user types: PRECO (pre-crypto officer for initial setup), CO (crypto officer — manages users), CU (crypto user — performs crypto operations). IAM controls management plane (create/delete cluster), HSM users control data plane.
- Clusters should have 2+ HSMs in different AZs for HA. Keys are automatically synchronized across HSMs in a cluster. If one HSM fails, the client SDK automatically fails over to another HSM. Single-HSM clusters have no failover.
- Backups are encrypted and stored in S3 (managed by AWS). Backups contain all HSM users and keys. Restoring a backup creates a new cluster with the same users and keys. Backups are automatic (daily) and can be triggered manually.
- PKCS#11, JCE, and OpenSSL integrations use the CloudHSM client SDK. Each has specific library paths and configuration. Version mismatches between SDK components cause failures.
Anti-hallucination rules
- Always cite specific cluster IDs, HSM IDs, or client log entries as evidence.
- AWS CANNOT recover keys or reset HSM users. Never suggest contacting AWS for key recovery.
- HSM users are NOT IAM users. Never confuse IAM and HSM user management.
- Port 2223-2225 is required for client connectivity. Never use other ports.
- Single-HSM clusters have NO failover. Never claim HA with one HSM.
- Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.
14 runbooks
| Category |
IDs |
Covers |
| A — Cluster |
A1–A2 |
Cluster creation, HSM provisioning |
| B — Connectivity |
B1–B2 |
Client connectivity, network issues |
| C — Keys |
C1–C2 |
Key management, key operations |
| D — Users |
D1–D2 |
User management, credential issues |
| E — Backup |
E1–E2 |
Backup/restore, backup failures |
| F — HA |
F1–F2 |
High availability, failover |
| G — SDK |
G1–G2 |
PKCS#11, JCE provider |
| H — Integration |
H1–H2 |
OpenSSL, SDK configuration |
| Z — Catch-All |
Z1 |
General troubleshooting |
1---2name: cloudhsm-diagnostics3description: Use this skill to investigate and troubleshoot AWS CloudHSM problems by analyzing cluster creation, HSM provisioning, client connectivity, key management, user management, backup/restore, high availability, CloudHSM SDK, PKCS#11, JCE provider, and OpenSSL integration following structured runbooks. Activate when: cluster creation failures, HSM initialization issues, client connection errors, key operation failures, crypto user management problems, backup failures, HA failover issues, SDK errors, PKCS#11 library problems, JCE provider exceptions, OpenSSL engine errors, or the user says something is wrong with CloudHSM without naming specific symptoms.4---56# AWS CloudHSM Diagnostics78## When to use910Any AWS CloudHSM investigation where the console alone is insufficient — cluster and HSM management, client connectivity, cryptographic key operations, HSM user management, backup and restore, high availability, SDK integration, PKCS#11 library, JCE provider, or OpenSSL engine issues.1112## Investigation workflow1314### Step 1 — Collect and triage1516```17aws cloudhsmv2 describe-clusters18aws cloudhsmv2 describe-clusters --filters clusterIds=<cluster-id>19aws cloudhsmv2 list-tags --resource-id <cluster-id>20```2122### Step 2 — Domain deep dive2324```25aws cloudhsmv2 describe-backups --filters clusterIds=<cluster-id>26aws ec2 describe-security-groups --group-ids <cluster-sg-id>27aws ec2 describe-subnets --subnet-ids <subnet-id>28# On client instance:29/opt/cloudhsm/bin/cloudhsm-cli cluster info30/opt/cloudhsm/bin/cloudhsm-cli user list31```3233### Step 3 — Detailed investigation3435```36aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=cloudhsmv2.amazonaws.com --max-results 2037aws cloudwatch get-metric-statistics --namespace AWS/CloudHSM --metric-name HsmKeysSessionCount --dimensions Name=ClusterId,Value=<id> --start-time <start> --end-time <end> --period 300 --statistics Average38# Check client logs:39# /opt/cloudhsm/run/cloudhsm-client.log40```4142Read `references/guardrails.md` before concluding on any CloudHSM issue.4344## Tool quick reference4546| Tool / API | When to use |47|------------|-------------|48| `describe-clusters` | Check cluster and HSM status |49| `describe-backups` | Check backup status and history |50| `cloudhsm-cli` | HSM-level user and key management |51| Security Groups | Verify client-to-HSM connectivity |52| CloudWatch Metrics | Monitor HSM health and usage |53| CloudTrail | Audit management API calls |54| Client logs | Debug connectivity and operation errors |5556## Gotchas: AWS CloudHSM5758- CloudHSM is a single-tenant HSM. You manage HSM users, keys, and access. AWS manages hardware and firmware. AWS cannot access your keys or recover them if credentials are lost. Losing all crypto officer (CO) credentials means losing key management access permanently.59- Client connectivity requires: (1) CloudHSM client installed on EC2 instance, (2) instance in same VPC or peered VPC, (3) security group allows port 2223-2225 from client to HSM ENI, (4) client configuration points to cluster IP.60- HSM users are managed INSIDE the HSM, not via IAM. Three user types: PRECO (pre-crypto officer for initial setup), CO (crypto officer — manages users), CU (crypto user — performs crypto operations). IAM controls management plane (create/delete cluster), HSM users control data plane.61- Clusters should have 2+ HSMs in different AZs for HA. Keys are automatically synchronized across HSMs in a cluster. If one HSM fails, the client SDK automatically fails over to another HSM. Single-HSM clusters have no failover.62- Backups are encrypted and stored in S3 (managed by AWS). Backups contain all HSM users and keys. Restoring a backup creates a new cluster with the same users and keys. Backups are automatic (daily) and can be triggered manually.63- PKCS#11, JCE, and OpenSSL integrations use the CloudHSM client SDK. Each has specific library paths and configuration. Version mismatches between SDK components cause failures.6465## Anti-hallucination rules66671. Always cite specific cluster IDs, HSM IDs, or client log entries as evidence.682. AWS CANNOT recover keys or reset HSM users. Never suggest contacting AWS for key recovery.693. HSM users are NOT IAM users. Never confuse IAM and HSM user management.704. Port 2223-2225 is required for client connectivity. Never use other ports.715. Single-HSM clusters have NO failover. Never claim HA with one HSM.726. Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.7374## 14 runbooks7576| Category | IDs | Covers |77|----------|-----|--------|78| A — Cluster | A1–A2 | Cluster creation, HSM provisioning |79| B — Connectivity | B1–B2 | Client connectivity, network issues |80| C — Keys | C1–C2 | Key management, key operations |81| D — Users | D1–D2 | User management, credential issues |82| E — Backup | E1–E2 | Backup/restore, backup failures |83| F — HA | F1–F2 | High availability, failover |84| G — SDK | G1–G2 | PKCS#11, JCE provider |85| H — Integration | H1–H2 | OpenSSL, SDK configuration |86| Z — Catch-All | Z1 | General troubleshooting |
aws-samples