Parallelcluster Diagnostics

ParallelCluster Diagnostics

When to use

Any AWS ParallelCluster investigation where the console alone is insufficient — cluster lifecycle management, compute fleet troubleshooting, Slurm scheduler debugging, shared storage issues, networking configuration, job execution problems, scaling behavior, or security configuration.

Investigation workflow

Step 1 — Collect and triage

pcluster list-clusters
pcluster describe-cluster --cluster-name <cluster-name>
pcluster get-cluster-stack-events --cluster-name <cluster-name>
pcluster describe-compute-fleet --cluster-name <cluster-name>
pcluster list-cluster-log-streams --cluster-name <cluster-name>

Step 2 — Domain deep dive

pcluster get-cluster-log-events --cluster-name <cluster-name> --log-stream-name <stream>
aws cloudformation describe-stack-events --stack-name <stack-name>
aws ec2 describe-instances --filters "Name=tag:parallelcluster:cluster-name,Values=<cluster-name>"
aws efs describe-file-systems --query 'FileSystems[?Tags[?Key==`parallelcluster:cluster-name` && Value==`<cluster-name>`]]'
aws fsx describe-file-systems

Step 3 — Detailed investigation

pcluster ssh --cluster-name <cluster-name>
# On head node:
sinfo
squeue
scontrol show nodes
scontrol show partitions
cat /var/log/parallelcluster/clustermgtd
cat /var/log/parallelcluster/slurm_resume.log
cat /var/log/parallelcluster/slurm_suspend.log

Read references/guardrails.md before concluding on any ParallelCluster issue.

Tool quick reference

Gotchas: ParallelCluster

• Cluster creation uses CloudFormation under the hood. Stack failures are the primary source of creation errors.
• Head node is a single EC2 instance — if it fails, the cluster is unavailable. Use placement groups and EBS snapshots for resilience.
• Slurm is the only supported scheduler in ParallelCluster 3.x. SGE and Torque were removed.
• Compute fleet uses dynamic nodes that are launched on demand by Slurm's power-saving plugin. Nodes are terminated after idle timeout.
• Shared storage options: EFS (elastic, multi-AZ), FSx for Lustre (high-performance), and EBS (head node only by default).
• FSx for Lustre scratch filesystems are temporary — data is lost when the cluster is deleted unless backed up to S3.
• Multi-queue support allows different instance types per Slurm partition. Each queue maps to a Slurm partition.
• Custom AMIs must be built with pcluster build-image. Using non-ParallelCluster AMIs causes failures.
• The head node needs internet access (or VPC endpoints) for package installation and AWS API calls.
• Auto-scaling is managed by clustermgtd and Slurm's power-saving plugin, not EC2 Auto Scaling groups.

Anti-hallucination rules

1. Always check CloudFormation stack events for cluster creation/update failures — they contain the actual error.
2. Never claim SGE or Torque are supported — ParallelCluster 3.x only supports Slurm.
3. FSx for Lustre scratch filesystems are NOT persistent — data is lost on cluster deletion.
4. Compute nodes are dynamic — they are launched by Slurm, not pre-provisioned.
5. Custom AMIs must be built with pcluster build-image — arbitrary AMIs will not work.
6. Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.

20 runbooks