Proxmox Ops

proxmox-ops

Judgment layer for Proxmox VE. The API gives you hands; this skill is the part that checks twice. Read operations are free. Write operations follow the golden rule. No exceptions.

Credentials

All scripts source ~/.config/claude-homelab/credentials.env if it exists, falling back to environment variables. Expected:

PROXMOX_HOST=192.168.1.10:8006        # host[:port]
PROXMOX_TOKEN_ID=claude@pve!claude    # user@realm!tokenname
PROXMOX_TOKEN_SECRET=uuid-secret
PROXMOX_VERIFY_TLS=false              # self-signed default; true if proper certs

API auth header: Authorization: PVEAPIToken=${PROXMOX_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}. Base URL: https://${PROXMOX_HOST}/api2/json.

Never use root@pam. Always a dedicated token with privilege separation: user claude@pve, role PVEAuditor on / for read-only, elevate later with PVEVMAdmin on a restricted path (e.g. /vms/<test-id>) for the write phase. If the configured token turns out to be root@pam, warn and recommend recreating it before doing anything else.

Missing credentials or 401: don't improvise — show the state, point to /homelab, stop. Token pasted in chat: warn (it's in history now), tell the user to delete the token in Datacenter → Permissions → API Tokens and re-run bash ~/.claude/skills/proxmox-ops/scripts/setup.sh in their own terminal.

Golden rule — write operations

Applies to: start/stop/shutdown/reset/delete of VM or CT, config changes, snapshot delete, storage changes, migration, restore. Before ANY of these:

1. Safety net when relevant: destructive ops (delete, restore, config change, snapshot delete) require a fresh snapshot or a verified recent backup (GET /nodes/{node}/storage/{storage}/content filtered on vzdump, check ctime). Start/stop don't need one, but check for locks first.
2. Show the exact operation: the full API call (method + path + body) or the exact shell command, before running it.
3. Explicit user confirmation on that exact operation. A yes to "restart the VM?" is not a yes to qm stop — show what will actually run.

Read operations (status, list, config read, task log read) are always free.

Async tasks (UPID) — poll before claiming success

Almost every Proxmox write returns immediately with a UPID string (UPID:node:...), NOT a result. The operation may still fail afterward.

Never announce success from the UPID alone. Always poll:

GET /nodes/{node}/tasks/{upid}/status   → repeat while "status": "running"

Terminal state: status: "stopped". Then check exitstatus: "OK" = success; anything else = failure — fetch the log:

GET /nodes/{node}/tasks/{upid}/log

and show the failing lines. Poll every 2s, timeout ~5 min for normal ops (longer for backup/restore/migration — tell the user it's long-running and keep polling).

Runbook: VM won't start

In order:

1. Task log first: the failed start created a task — GET /nodes/{node}/tasks?vmid={vmid}&limit=5, read the error of the most recent qmstart. The error usually names the culprit directly.
2. Config: GET /nodes/{node}/qemu/{vmid}/config — look for missing ISO in cdrom drive, hostpci device that disappeared, bridge that doesn't exist (vmbr typo), memory > host capacity.
3. Storage: is the disk's storage active? GET /nodes/{node}/storage → active: 1? Full? Volume actually present (pvesm list <storage> --vmid <vmid>)?
4. Locks: lock: field in config (backup, snapshot, migrate). A stale lock after an interrupted backup is classic. Clearing it (qm unlock <vmid>) is a WRITE op → golden rule: confirm first, and only after verifying no backup/migration is genuinely running (GET /nodes/{node}/tasks?running=1).

Runbook: storage full

Identify the consumers, biggest first:

1. Overview: GET /nodes/{node}/storage — which store, how full.
2. Orphan disks: pvesm list <storage> and compare vmid column against existing guests — volumes of deleted guests, and unused0/unused1 entries in guest configs.
3. Backup accumulation: vzdump files beyond retention — GET /nodes/{node}/storage/{storage}/content?content=backup, sort by ctime, compare against retention settings (prune-backups).
4. Old templates/ISOs: content=iso,vztmpl.
5. Snapshot sprawl: guests with many/old snapshots (GET .../qemu/{vmid}/snapshot); qcow2/ZFS snapshots grow with delta.
6. Deleting any of it = write op → golden rule. Propose a deletion list with sizes and let the user pick.

Runbook: backup & migration (vzdump)

• Backup one guest: vzdump <vmid> --storage <store> --mode snapshot --compress zstd. Mode snapshot = minimal downtime (needs snapshot-capable storage); suspend = pause; stop = clean shutdown, most consistent.
• Restore: qmrestore <archive> <newvmid> / pct restore — restoring OVER an existing vmid destroys it → golden rule, backup check, confirm.
• Migration: online needs shared storage or --with-local-disks (block-copies, slow). Check target node capacity first (GET /nodes/{target}/status). It's a task: poll the UPID.
• Failed backups: task log; classic causes = storage full, guest lock, qemu-guest-agent absent while --mode snapshot with fsfreeze expected.

Runbook: LXC unprivileged vs privileged

Default and recommendation: unprivileged (unprivileged: 1). Root in container = uid 100000 on host.

uid/gid mapping traps:

• Bind mounts: host files owned by uid 1000 appear as nobody in the container. Fix: chown host-side to 101000 (100000 + container uid), or custom idmap in /etc/pve/lxc/<id>.conf (lxc.idmap entries + matching /etc/subuid, /etc/subgid on the host).
• NFS/CIFS mounts inside the CT: blocked in unprivileged. Mount on the host, bind-mount into the CT (mp0: /host/path,mp=/ct/path).
• Docker inside LXC: needs features: nesting=1 (+ keyctl=1); works unprivileged on recent PVE, but a VM is the more robust choice.
• Privileged only when a device/kernel feature genuinely requires it — say why when recommending it.

Pattern: create LXC from template

1. Template present? pvesm list <store> --content vztmpl; else pveam update && pveam available --section system then pveam download <store> <template>.
2. Recommended create (adjust sizes to need):

pct create <vmid> <store>:vztmpl/<template> \
  --hostname <name> --unprivileged 1 \
  --cores 2 --memory 2048 --swap 512 \
  --rootfs <store>:8 \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp,firewall=1 \
  --features nesting=1 \
  --start 0

3. It's a write op → golden rule (show command, confirm). Then start, poll the UPID, and verify with pct exec <vmid> -- ip a (SSH fallback).

Pattern: cluster config changes

Before ANY modification under /etc/pve (storage.cfg, corosync, etc.): tar czf /root/pve-backup-$(date +%F-%H%M).tar.gz /etc/pve (via SSH — /etc/pve is a fuse mount of pmxcfs, snapshots don't cover it). Single-node "cluster": no quorum issues; in a real cluster, never edit corosync.conf without quorum understanding — see references.

GPU/device passthrough (basic)

Precondition: IOMMU on (dmesg | grep -i iommu), device in its own IOMMU group. VM must be machine: q35; add hostpci0: <bus:dev.fn>,pcie=1. Host must not claim the device (blacklist driver or vfio-pci early bind). Config change = write op → golden rule + snapshot/backup first.

API vs SSH fallback

Prefer API always — auditable, token-scoped, no shell risk. SSH needed for:

• pct exec / commands inside a CT (no API endpoint)
• /etc/pve backup (fuse mount)
• qm unlock when API refuses due to the lock itself
• pveam template downloads (API partial)
• Reading host files (sysctl, /etc/network/interfaces)

When falling back to SSH, show the exact command and say why the API can't do it. SSH commands respect the golden rule too.

References & scripts

• references/cli-cheatsheet.md — qm/pct/pvesm/vzdump flags that matter, known traps (locks, single-node quorum, storage content types).
• scripts/health-check.sh — nodes, storage, guests, recent failed tasks via API. Read-only, safe anytime.
• scripts/setup.sh — credential onboarding; run by the USER in their terminal, never by Claude.

Test rules (when operating on a real cluster)

• Phase 1: READ ONLY (status, list, health-check).
• Phase 2 (only after the user's explicit GO): writes ONLY on the designated test VM/CT — never anything else.