# Parallelcluster Diagnostics

> Use this skill to investigate and troubleshoot AWS ParallelCluster problems by analyzing cluster creation, compute fleet, Slurm scheduler, shared storage, networking, job execution, auto-scaling, and security configurations using structured runbooks. Activate when: cluster creation failures, update issues, deletion problems, compute fleet errors, Slurm scheduler issues, shared storage (EFS/FSx) problems, scratch storage failures, VPC configuration issues, multi-AZ problems, job submission errors, job failures, auto-scaling issues, capacity problems, IAM permission errors, SSH access issues, or the user says something is wrong with ParallelCluster without naming specific symptoms.

- Skill: `aws-samples/parallelcluster-diagnostics` (Agent Skill, multi-file: 20 files)
- Install (CLI): `npx skillmds@latest add aws-samples/parallelcluster-diagnostics`
- Raw SKILL.md: https://api.skillmd.com/api/skills/aws-samples/parallelcluster-diagnostics/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: aws-samples (https://skillmd.com/u/aws-samples)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/aws-samples/parallelcluster-diagnostics

---


# ParallelCluster Diagnostics

## When to use

Any AWS ParallelCluster investigation where the console alone is insufficient — cluster lifecycle management, compute fleet troubleshooting, Slurm scheduler debugging, shared storage issues, networking configuration, job execution problems, scaling behavior, or security configuration.

## Investigation workflow

### Step 1 — Collect and triage

```
pcluster list-clusters
pcluster describe-cluster --cluster-name <cluster-name>
pcluster get-cluster-stack-events --cluster-name <cluster-name>
pcluster describe-compute-fleet --cluster-name <cluster-name>
pcluster list-cluster-log-streams --cluster-name <cluster-name>
```

### Step 2 — Domain deep dive

```
pcluster get-cluster-log-events --cluster-name <cluster-name> --log-stream-name <stream>
aws cloudformation describe-stack-events --stack-name <stack-name>
aws ec2 describe-instances --filters "Name=tag:parallelcluster:cluster-name,Values=<cluster-name>"
aws efs describe-file-systems --query 'FileSystems[?Tags[?Key==`parallelcluster:cluster-name` && Value==`<cluster-name>`]]'
aws fsx describe-file-systems
```

### Step 3 — Detailed investigation

```
pcluster ssh --cluster-name <cluster-name>
# On head node:
sinfo
squeue
scontrol show nodes
scontrol show partitions
cat /var/log/parallelcluster/clustermgtd
cat /var/log/parallelcluster/slurm_resume.log
cat /var/log/parallelcluster/slurm_suspend.log
```

Read `references/guardrails.md` before concluding on any ParallelCluster issue.

## Tool quick reference

| Tool / API | When to use |
|------------|-------------|
| `pcluster describe-cluster` | Cluster status and config |
| `pcluster get-cluster-stack-events` | CloudFormation events |
| `pcluster describe-compute-fleet` | Compute fleet status |
| `pcluster list-cluster-log-streams` | Available log streams |
| `pcluster get-cluster-log-events` | Log content |
| `sinfo` | Slurm node/partition status |
| `squeue` | Job queue status |
| `scontrol show nodes` | Detailed node info |
| `sacct` | Job accounting data |
| `cloudformation describe-stack-events` | Stack events |

## Gotchas: ParallelCluster

- Cluster creation uses CloudFormation under the hood. Stack failures are the primary source of creation errors.
- Head node is a single EC2 instance — if it fails, the cluster is unavailable. Use placement groups and EBS snapshots for resilience.
- Slurm is the only supported scheduler in ParallelCluster 3.x. SGE and Torque were removed.
- Compute fleet uses dynamic nodes that are launched on demand by Slurm's power-saving plugin. Nodes are terminated after idle timeout.
- Shared storage options: EFS (elastic, multi-AZ), FSx for Lustre (high-performance), and EBS (head node only by default).
- FSx for Lustre scratch filesystems are temporary — data is lost when the cluster is deleted unless backed up to S3.
- Multi-queue support allows different instance types per Slurm partition. Each queue maps to a Slurm partition.
- Custom AMIs must be built with pcluster build-image. Using non-ParallelCluster AMIs causes failures.
- The head node needs internet access (or VPC endpoints) for package installation and AWS API calls.
- Auto-scaling is managed by clustermgtd and Slurm's power-saving plugin, not EC2 Auto Scaling groups.

## Anti-hallucination rules

1. Always check CloudFormation stack events for cluster creation/update failures — they contain the actual error.
2. Never claim SGE or Torque are supported — ParallelCluster 3.x only supports Slurm.
3. FSx for Lustre scratch filesystems are NOT persistent — data is lost on cluster deletion.
4. Compute nodes are dynamic — they are launched by Slurm, not pre-provisioned.
5. Custom AMIs must be built with pcluster build-image — arbitrary AMIs will not work.
6. Spend no more than 2 minutes on any single hypothesis. Pivot if inconclusive.

## 20 runbooks

| Category | IDs | Covers |
|----------|-----|--------|
| A — Cluster | A1-A3 | Creation failures, update, deletion |
| B — Compute | B1-B2 | Compute fleet, Slurm scheduler |
| C — Storage | C1-C2 | Shared storage (EFS/FSx), scratch |
| D — Networking | D1-D2 | VPC config, multi-AZ |
| E — Jobs | E1-E2 | Job submission, job failures |
| F — Scaling | F1-F2 | Auto-scaling, capacity |
| G — Security | G1-G2 | IAM, SSH access |
| Z — Catch-All | Z1 | General troubleshooting |