# Vps Provisioning

> VPS Provisioning

- Skill: `lucadominguez/vps-provisioning` (Agent Skill)
- Install (CLI): `npx skillmds@latest add lucadominguez/vps-provisioning`
- Raw SKILL.md: https://api.skillmd.com/api/skills/lucadominguez/vps-provisioning/raw
- Safety review: WARNING
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: lucadominguez (https://skillmd.com/u/lucadominguez)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/lucadominguez/vps-provisioning

---

# VPS Provisioning

Bootstrap a fresh Linux VPS from zero to production-ready foundation. Used when the user says "set up my VPS," "provision my server," "bootstrap a new droplet/instance," or provides a provisioning checklist.

## Trigger conditions

- User asks to set up, provision, or bootstrap a VPS/server/instance
- User provides an IP address with provisioning instructions
- User mentions "Phase 0" or similar foundation-laying tasks for a server

## Prerequisites

Before starting any provisioning work:

1. **Verify OS** against requirements. If the OS doesn't match (e.g., AlmaLinux instead of Ubuntu), STOP and ask the operator. Do not adapt on the fly — later phases depend on OS-specific scripts.
2. **Confirm credentials work** — test SSH connectivity before doing anything else.
3. **Check if the VPS is fresh** — existing Docker containers, data in `/data/`, or prior installs mean stop and report.

## SSH connection troubleshooting

If `ssh` hangs during key exchange (no output at all, even with `-v`), the server likely has broken default cipher/KEX negotiation. This is common on some VPS providers after OS reinstall.

### Diagnostic: confirm the hang is KEX-related

```bash
# Netcat should show the SSH banner instantly
echo "QUIT" | nc $IP 22
# If banner shows but ssh hangs, it's a KEX/cipher negotiation issue
```

### Fix: explicit cipher and KEX selection

```bash
ssh \
  -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null \
  -o ConnectTimeout=10 \
  -o KexAlgorithms=curve25519-sha256 \
  -o HostKeyAlgorithms=ssh-ed25519 \
  -c aes128-ctr \
  -m hmac-sha2-256 \
  root@$IP 'echo OK'
```

These options force a minimal, widely-supported algorithm set that avoids negotiation hangs. Store this as a shell alias or function when working with a problematic host.

### Alternative: SSH config snippet for problematic hosts

```
Host problematic-vps
    HostName 38.49.217.154
    KexAlgorithms curve25519-sha256
    HostKeyAlgorithms ssh-ed25519
    Ciphers aes128-ctr
    MACs hmac-sha2-256
```

### Password auth with sshpass

When key auth isn't set up yet (fresh VPS, only root password available):

```bash
# Install if needed
sudo apt-get install -y sshpass

# Use with single quotes to protect special characters
sshpass -p 'password' ssh <user>@<host> 'command'

# For passwords with tricky characters, use a temp file
printf '%s' 'password' > /tmp/pass.txt
sshpass -f /tmp/pass.txt ssh <user>@<host> 'command'
rm /tmp/pass.txt
```

### Auth debugging

If password is rejected, check what methods the server accepts:

```bash
ssh -v -o PreferredAuthentications=publickey,password <user>@<host> 2>&1 | \
  grep -E 'Authentications that can continue|permission|denied|method'
```

Key lines:
- `Authentications that can continue: publickey,password` — password auth is enabled
- `Authentications that can continue: publickey` — password is disabled, need key
- `we sent a password packet, wait for reply` then `Permission denied` — wrong password

### Password auth troubleshooting

- Ubuntu 24.04 cloud images often lock the root account — password auth works on console but not over SSH.
- If `sshpass` with correct password returns "Permission denied," check `PermitRootLogin` and `PasswordAuthentication` in `/etc/ssh/sshd_config`.
- Use `ssh -v` and grep for `Authentications that can continue` to see what methods the server offers.
- **Backtick gotcha**: Backtick characters (`` ` ``) in chat messages are often markdown formatting, not part of the actual password. If a password is shared like `` the password is `abc`123 `` — test with AND without the backticks.
- **fail2ban**: Don't loop on wrong password — after 2-3 failed attempts, stop and ask. Many hosts have fail2ban that will block your IP.

### sshd vs ssh service name (Ubuntu 24.04)

On Ubuntu 24.04, the SSH service is named `ssh`, not `sshd`. If `systemctl reload sshd` fails with "Unit sshd.service not found", use `systemctl reload ssh` instead.

### Output-size hang with custom cipher config

On servers that require explicit KEX/cipher flags (see "SSH connection troubleshooting" above), command output exceeding approximately **800 bytes hangs indefinitely**. This affects `cat`, `head`, `sed`, and Python printing — any command producing more than a trivial amount of output.

**Symptom**: `wc -l <file>` works (output is small), but `cat <file>` or `head -50 <file>` hangs/timeouts on the same file. `whoami`, `echo OK`, and other tiny-output commands work fine. The cutoff appears around 800–1000 bytes.

**Workarounds**:
1. Check file size first: `wc -c <file>` — if under ~700 bytes, safe to cat directly.
2. Read in small chunks: `sed -n '1,10p' <file>`, `sed -n '11,20p' <file>`, etc. Keep each chunk under 700 bytes.
3. Use commands with inherently small output: `xxd <file> | head -5`, `file <file>`, `wc -l <file>`.
4. For configuration files, prefer `grep` for specific keys rather than reading the whole file.
5. For long transcripts/doctor output, grep for status lines: `gbrain doctor 2>&1 | grep -E '\[(OK|WARN|FAIL)\]'`

**Root cause**: the `aes128-ctr` cipher + explicit KEX settings create a packet-fragmentation issue when the SSH channel sends more than ~800 bytes in a single burst. This is NOT a general SSH issue — it only manifests on hosts that need the explicit cipher workaround. Normal SSH connections without custom cipher config don't have this problem.

### File transfer (scp) with custom cipher hosts

When `scp` is needed to transfer files to a host that requires explicit cipher/KEX flags, use `-o` for all options — the `scp` command does NOT support the `-m` flag (unlike `ssh`):

```bash
scp -o KexAlgorithms=curve25519-sha256 \
    -o HostKeyAlgorithms=ssh-ed25519 \
    -c aes128-ctr \
    -o MACs=hmac-sha2-256 \
    local-file vps:/remote/path
```

Note: `-o MACs=` (not `-m`), `-c` for cipher, and all other flags via `-o`.

## Tailscale Setup

```bash
curl -fsSL https://tailscale.com/install.sh | sh
tailscale up
```

`tailscale up` prints a one-time auth URL and hangs waiting for browser authentication. Run it as a **background process** while the user authenticates in their browser.

### Auth URL behavior

- The URL changes **every time** `tailscale up` is restarted — old URLs expire immediately
- If the user doesn't authenticate in time, kill the process and re-run `tailscale up` to get a fresh URL
- Running `tailscale status` while `tailscale up` is waiting shows `Logged out.` + a potentially different URL — **ignore this**. Only the URL from the running `tailscale up` process matters
- Once authenticated, `tailscale status` shows the node as online with its tailnet IP
- Free tier: 100 devices, 3 users — sufficient for single-VPS builds

### Pitfall

Multiple `tailscale up` processes will conflict. Before restarting, kill any existing `tailscale up` process first.

## SSH key bootstrap via VNC/console

When password auth is locked but the VPS provider offers a web console (VNC):

1. Generate an SSH key locally (if not already present):
   ```bash
   ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ""
   ```

2. Give the operator your public key:
   ```bash
   cat ~/.ssh/id_ed25519.pub
   ```

3. Operator runs these commands ONE AT A TIME in the VNC console (do NOT combine with `&&` — paste each line separately):

   First, create the directory:
   ```bash
   mkdir -p /root/.ssh
   ```

   Then write the key. Use **double quotes** (`"`) around the key — single quotes can cause the file to be empty on some VNC consoles:
   ```bash
   echo "ssh-ed25519 AAAAC3NzaC1l... user@host" > /root/.ssh/authorized_keys
   ```

   Then verify the key was written:
   ```bash
   cat /root/.ssh/authorized_keys
   ```

4. Fix permissions (CRITICAL — SSH silently rejects keys with wrong perms):
   ```bash
   chmod 700 /root/.ssh
   chmod 600 /root/.ssh/authorized_keys
   ```

**Pitfall — bare paste**: If the operator pastes the key without the `echo "..."` wrapper, the shell will try to execute it as a command and fail with "ssh-ed... command not found." Always provide the full `echo "..."` line.

**Pitfall — quoting**: Use **double quotes** (`echo "key"`), not single quotes (`echo 'key'`). On some VNC/web consoles, single-quoted strings get mangled during copy-paste and produce empty files. If `cat /root/.ssh/authorized_keys` shows nothing, re-run with double quotes.

**Pitfall — `>` vs `>>`**: Use `>` (overwrite) when setting up the key for the first time — this is idempotent on re-run. Only use `>>` (append) if you are intentionally adding a second key to an existing file. Using `>>` on repeated failed attempts duplicates the key line.

**Pitfall — permissions**: Even with the key correctly written, SSH will reject it if `/root/.ssh` is not mode `700` or `authorized_keys` is not mode `600`. Always run the `chmod` commands and verify.

## Provisioning checklist pattern

When given a structured provisioning plan (Phase 0, Phase 1, etc.):

- Respect phase boundaries — don't start Phase N+1 work even if it "seems obvious"
- Track progress in a status file (e.g., `/var/log/hermes/phase-0-status.json`) so re-runs pick up where they left off
- Each step should be idempotent: check if work is already done before doing it
- Log all commands to `/var/log/hermes/actions.log` with timestamp, command, exit code, and rationale

### Resumption audit (when continuing a partially-done provision)

When resuming a provision started in a prior session, do NOT trust the status file alone — prior runs may have left it sparse or empty. Run a **bulk audit** first: one SSH command that checks all key indicators at once (OS, hostname, users + groups, SSH config, UFW status, Tailscale status, Docker version + compose + daemon.json existence, /data/ tree, /srv/stack/, restic, cron entries, actions log). This 5–10 second check gives a definitive picture of what's done vs pending without guessing from an unreliable status file.

### SSH verification: test from the agent's machine, not localhost

When verifying non-root SSH access (Phase 0.2 / 0.9), do NOT test with `sudo -u <user> ssh localhost` from the VPS — this triggers localhost key prompts and may fail even when everything is correct. The real verification is an SSH connection **from the agent's machine** (WSL, your workstation) directly to the VPS as the non-root user. If the VPS requires explicit KEX/cipher flags (see SSH troubleshooting above), apply those flags to the test command too.

## Typical Phase 0 tasks (Ubuntu 24.04)

1. System baseline: OS confirm, apt update/upgrade, hostname, timezone UTC, unattended-upgrades (security only)
2. User and SSH: non-root sudo user, key-only SSH, docker group
3. Firewall: UFW (deny incoming, allow outgoing, allow SSH 22), Tailscale
4. Docker: Engine + Compose v2 from official repo, journald log driver, ulimit config. **Critical**: Docker installs often complete without `/etc/docker/daemon.json` — always check for it and create it explicitly. Use `templates/docker-daemon.json` as the starting point. After writing, run `systemctl reload docker`.
   **Pitfall — host access to container databases**: When exposing a Dockerized Postgres on a host port (`127.0.0.1:5432:5432`), connections from the host to the container arrive from the Docker bridge IP (typically `172.x.0.1`), not from `127.0.0.1`. If `pg_hba.conf` only allows local loopback, connections will fail with `FATAL: no pg_hba.conf entry for host "172.18.0.1"`. Fix: set `POSTGRES_HOST_AUTH_METHOD=trust` in the container environment for local-only setups (the container is already bound to `127.0.0.1` on the host side, so remote hosts cannot reach it).
   **Pitfall — PostgreSQL config quoting**: In `postgresql.conf`, string values use **single quotes**, never double quotes. `listen_addresses = "*"` (double quotes) causes a syntax error and postgres won't start. Use `listen_addresses = '*'` (single quotes). Double quotes in PostgreSQL config are for identifiers, not values.
5. Directory layout: `/data/` tree (raw, processed, brain, postgres, redis, minio, logs), `/srv/stack/`
6. Backups: restic + Backblaze B2, backup script, cron. **B2 is optional** — the operator may decline it. If declined, skip restic initialization, mark the task as skipped in the status file, and note that pg_dump-only local backups should be added once databases exist (Phase 1+).
7. Health monitoring: healthcheck.sh (disk, memory, load, containers), cron. See `templates/healthcheck.sh` for a working template. **Pitfall**: avoid `bc` for floating-point comparisons in bash healthcheck scripts — `bc -l` in subshells can hang silently on some systems. Use integer math (`cut -d. -f1` for load, integer division for percentages) instead.
8. Self-config: agent state directory, phase tracking, secrets
9. Final verification: run all checks, report summary

See `references/phase-0-template.md` for the full detailed task list.
See `references/gbrain-install.md` for gbrain setup (Phase 1 knowledge-brain stack).