# Debug Node

> Use when a Hetzner node is unreachable, SSH fails, cloud-init seems broken, or provisioning hangs. Boots into rescue mode via hcloud CLI to inspect filesystem, logs, SSH keys, sshd config, and cloud-init state without needing SSH access to the node itself.

- Skill: `mysticaltech/debug-node` (Agent Skill)
- Install (CLI): `npx skillmds@latest add mysticaltech/debug-node`
- Raw SKILL.md: https://api.skillmd.com/api/skills/mysticaltech/debug-node/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: mysticaltech (https://skillmd.com/u/mysticaltech)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/mysticaltech/debug-node

---


# Debug Hetzner Node via Rescue Console

## Overview

When a Hetzner Cloud server is unreachable (SSH hangs, provisioning stuck, cloud-init failure), this skill uses Hetzner's **rescue mode** to mount the node's filesystem and inspect everything from the outside — no working SSH required.

## Usage

```
/debug-node
```

When invoked, ask for:
1. The server name or IP (can be found from `hcloud server list`)
2. What symptom they're seeing (SSH timeout, provisioning hang, etc.)

## Prerequisites

- `hcloud` CLI installed and configured with a valid token
- The server must exist in Hetzner Cloud

```bash
hcloud server list
```

## Leap Micro Filesystem Model

Leap Micro uses a **transactional-update** system on btrfs. This is the mental model for everything below.

| Layer | Writable? | Persists reboot? | Persists Hetzner snapshot? |
|-------|-----------|------------------|---------------------------|
| `/usr` (snapshot) | No (read-only) | Yes | Yes |
| `/etc` via `transactional-update shell` | Yes (new snapshot) | Yes (after reboot) | Yes |
| `/etc` via direct edit on running system | Yes (current snapshot overlay) | Yes, unless a pending snapshot changed the same path | Current-snapshot content only |
| `/var` (separate subvolume) | Yes | Yes | Yes |

**Rule:** Ordinary `/etc` edits persist across a normal reboot. However, after `transactional-update` creates a pending snapshot, concurrent edits to the current snapshot's `/etc` are not merged when the pending snapshot changed the same path. Put image-building and package-coupled `/etc` changes inside `transactional-update --continue shell`, or reboot into the pending snapshot before editing.

**Packer build phases:**
1. **Rescue mode:** Write qcow2 to disk, reboot
2. **`install_packages`:** Inside `transactional-update` — changes **persist**
3. **`clean_up`:** Volatile overlay — `/etc` changes are **lost** in the Hetzner snapshot

## Step 1: Identify the Server

```bash
hcloud server list -o columns=id,name,status,ipv4 | grep <pattern>
```

## Step 2: Enable Rescue Mode & Reboot

```bash
hcloud server enable-rescue <SERVER_ID> --type linux64
hcloud server reboot <SERVER_ID>
sleep 30
```

Save the rescue root password from the output (usually key auth works, but just in case).

## Step 3: SSH into Rescue

```bash
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@<SERVER_IP>
```

## Step 4: Mount the Filesystem

### Leap Micro / MicroOS (btrfs)

```bash
# Mount btrfs top-level
mount -o subvolid=5 /dev/sda3 /mnt

# List snapshots — highest number is active
ls /mnt/@/.snapshots/
```

Layout:
```
/mnt/@/.snapshots/N/snapshot/    latest active snapshot — /etc lives here
/mnt/@/root/                     /root home (/root/.ssh/authorized_keys)
/mnt/@/var/                      /var (logs, cloud-init state, journal)
```

**Key:** `/etc` is inside the snapshot. `/var` and `/root` are separate subvolumes at `@/var` and `@/root`.

### Unsupported/custom ext4 images

```bash
mount /dev/sda1 /mnt
```

## Step 5: Diagnostic Checklist

Set this once and use throughout:
```bash
SNAP=/mnt/@/.snapshots/N/snapshot   # replace N with highest snapshot number
```

### 5a. Cloud-Init Status

Start here — most provisioning failures trace back to cloud-init.

```bash
cat /mnt/@/var/lib/cloud/data/result.json
cat /mnt/@/var/lib/cloud/data/status.json
cat /mnt/@/var/lib/cloud/instance/datasource

# What Terraform actually sent
cat /mnt/@/var/lib/cloud/instance/user-data.txt
zcat /mnt/@/var/lib/cloud/instance/user-data.txt.i 2>/dev/null

# Logs
tail -100 /mnt/@/var/log/cloud-init.log
tail -100 /mnt/@/var/log/cloud-init-output.log
```

**Expected:** `DataSourceHetzner`, no errors.
**Watch for:** `Skipping modules` — means cloud-init already ran for this instance-id.

Cloud-init facts on Hetzner + Leap Micro:
- Datasource: `DataSourceHetzner` (metadata API)
- Terraform's `cloudinit_config` → gzip+base64 multipart MIME → `user_data`
- `disable_root: false` prevents cloud-init from disabling root but does NOT unlock a locked account
- `ssh_authorized_keys` writes keys to `/root/.ssh/authorized_keys`

### 5b. SSH Keys

```bash
cat /mnt/@/root/.ssh/authorized_keys
```

Compare with your local pubkey. If missing, cloud-init failed to inject — check 5a logs.

### 5c. SSHD Configuration

Config loading order (first match wins):

```
1. /etc/ssh/sshd_config.d/40-kube-hetzner-authorized-keys-command.conf
2. /etc/ssh/sshd_config.d/50-cloud-init.conf
3. /etc/ssh/sshd_config.d/kube-hetzner.conf           (MaxAuthTries 2)
4. /usr/etc/ssh/sshd_config.d/40-suse-crypto-policies.conf
5. /usr/etc/ssh/sshd_config                            (UsePAM yes)
```

```bash
ls $SNAP/etc/ssh/sshd_config.d/
cat $SNAP/etc/ssh/sshd_config.d/*.conf
cat $SNAP/usr/etc/ssh/sshd_config
ls -la $SNAP/etc/ssh/ssh_host_*
```

### 5d. Account Status

```bash
grep '^root:' $SNAP/etc/shadow
```

| Pattern | Meaning | SSH pubkey works? |
|---------|---------|-------------------|
| `root:*:...` | Unlocked, no password | Yes |
| `root:!*:...` or `root:!:...` | Locked | **No** (PAM rejects with `UsePAM yes`) |

This is fixed in packer (`usermod -p '*' root` inside transactional-update) with a cloud-init `bootcmd` safety net. If you see a locked account on a fresh node, the packer snapshot needs rebuilding.

### 5e. Journal Logs

```bash
journalctl -D /mnt/@/var/log/journal/ -u sshd --no-pager | tail -50
journalctl -D /mnt/@/var/log/journal/ -u k3s --no-pager | tail -30
journalctl -D /mnt/@/var/log/journal/ -u rke2-server --no-pager | tail -30
journalctl -D /mnt/@/var/log/journal/ -u rke2-agent --no-pager | tail -30
```

### 5f. Network

```bash
ls $SNAP/etc/NetworkManager/system-connections/
cat $SNAP/etc/NetworkManager/system-connections/*.nmconnection 2>/dev/null
```

### 5g. Kubernetes

```bash
cat $SNAP/etc/rancher/k3s/config.yaml 2>/dev/null
cat $SNAP/etc/rancher/rke2/config.yaml 2>/dev/null
cat /mnt/@/var/lib/rancher/k3s/server/token 2>/dev/null
```

### 5h. SELinux

```bash
cat $SNAP/etc/selinux/config
chroot $SNAP rpm -qa | grep -iE 'selinux|k3s|rke2'
grep -i 'avc:.*denied' /mnt/@/var/log/audit/audit.log | tail -50
journalctl -D /mnt/@/var/log/journal/ --no-pager | grep -i 'avc:.*denied' | tail -50
```

For workload denials, follow `docs/selinux.md`: collect the AVC lines,
workload name/version, k3s/RKE2 distribution, OS image, and udica result before
proposing upstream policy changes. Do not globally disable SELinux as the first
answer; use per-pool `selinux = false` only as the last resort for a workload or
nodepool that cannot run under policy.

## Step 6: Apply a Fix

Edit files in the **active snapshot** (`$SNAP`), not in `@/` base.

```bash
# Unlock root account (if locked)
sed -i 's/^root:!*/root:*/' $SNAP/etc/shadow

# Fix authorized_keys
mkdir -p /mnt/@/root/.ssh
echo "ssh-ed25519 AAAA..." > /mnt/@/root/.ssh/authorized_keys
chmod 700 /mnt/@/root/.ssh && chmod 600 /mnt/@/root/.ssh/authorized_keys

# Regenerate host keys
mount --bind /proc $SNAP/proc && mount --bind /sys $SNAP/sys && mount --bind /dev $SNAP/dev
chroot $SNAP ssh-keygen -A
umount $SNAP/proc $SNAP/sys $SNAP/dev
```

**Note:** Rescue-mode edits are immediate fixes. The proper long-term fix belongs in the packer template or cloud-init.

## Step 7: Reboot to Normal

```bash
umount /mnt 2>/dev/null
exit
```

```bash
hcloud server disable-rescue <SERVER_ID>
hcloud server reboot <SERVER_ID>
sleep 60
ssh -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 root@<SERVER_IP> 'echo ok'
```

## Common Diagnoses

| Symptom | Likely Cause | Check | Fix |
|---------|-------------|-------|-----|
| SSH timeout | Firewall or network | Hetzner firewall rules | Open port 22 |
| SSH "Connection refused" | sshd not running | Journal logs | Fix sshd config syntax |
| SSH key rejected | Keys not injected | `authorized_keys` empty | Check cloud-init logs |
| SSH "Too many auth failures" | Agent offers too many keys | `MaxAuthTries 2` | Use `-o IdentitiesOnly=yes` |
| SSH "unable to authenticate" | Root locked, or key mismatch | `/etc/shadow`, authorized_keys | Rebuild packer snapshot |
| Provisioner hangs "Still creating" | SSH can't connect | All above | Fix underlying SSH issue |
| Cloud-init skips modules | Already ran for instance-id | cloud-init.log | Clean `/var/lib/cloud/instance` |
| k3s/rke2 not starting | Config or SELinux | Journal + audit.log | Fix config or policy |
| Workload denied by SELinux | Missing workload policy | AVC lines in audit/journal | Follow `docs/selinux.md`; try udica before disabling a pool |
| Network/subnet destroy hangs | Autoscaler-created server outside Terraform state | `hcloud server list` for cluster-name or `kh-ci-*` leftovers | Delete only after control plane is dead, or scale autoscaler `min_nodes = 0` first |
| `/etc` change vanished after transactional reboot | Current and pending snapshots changed the same path | Inspect `transactional-update status` and `/var/lib/overlay` | Reapply in the active snapshot or move image-build changes into the transaction |

## Debugging SSH Manually

```bash
# Verbose with specific key (avoids agent key spray hitting MaxAuthTries 2)
ssh -vvv -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 root@<SERVER_IP>

# In -vvv output:
# "Offering public key: ..."              → key was offered
# "Server accepts key: ..."               → success path
# "Authentications that can continue: ..."  → key was REJECTED
# "Too many authentication failures"       → agent sent too many keys
```

## Pro Tips

1. **Mount with `subvolid=5`** — gets the real btrfs root, navigate to `@/.snapshots/N/snapshot/`
2. **Highest snapshot = active** — that's where `/etc` lives
3. **`/var` is separate** — logs and cloud-init are at `/mnt/@/var/`, not inside the snapshot
4. **Journal without a running system** — `journalctl -D /path/to/journal/`
5. **Use `-o IdentitiesOnly=yes`** — kube-hetzner sets `MaxAuthTries 2`
6. **Volatile overlay trap** — if rescue shows different content than the live system did, it was running on a volatile overlay that never got committed
7. **After fixing packer, rebuild snapshots** — verify build logs show changes inside the `transactional-update` output
8. **Rescue mode is non-destructive** — you're just reading/writing files on the disk
9. **Destroy with the wrapper** — for full-cluster teardown, run `scripts/destroy.sh` from the Terraform root; it retries only the known ingress-LB detach race and prints a read-only orphan report. Use `scripts/cleanup.sh` only as the forceful fallback; it treats the token's entire HCloud project as cluster-dedicated, so review the dry run first.