Vps Provisioning

VPS Provisioning

Bootstrap a fresh Linux VPS from zero to production-ready foundation. Used when the user says "set up my VPS," "provision my server," "bootstrap a new droplet/instance," or provides a provisioning checklist.

Trigger conditions

• User asks to set up, provision, or bootstrap a VPS/server/instance
• User provides an IP address with provisioning instructions
• User mentions "Phase 0" or similar foundation-laying tasks for a server

Prerequisites

Before starting any provisioning work:

1. Verify OS against requirements. If the OS doesn't match (e.g., AlmaLinux instead of Ubuntu), STOP and ask the operator. Do not adapt on the fly — later phases depend on OS-specific scripts.
2. Confirm credentials work — test SSH connectivity before doing anything else.
3. Check if the VPS is fresh — existing Docker containers, data in /data/, or prior installs mean stop and report.

SSH connection troubleshooting

If ssh hangs during key exchange (no output at all, even with -v), the server likely has broken default cipher/KEX negotiation. This is common on some VPS providers after OS reinstall.

Diagnostic: confirm the hang is KEX-related

# Netcat should show the SSH banner instantly
echo "QUIT" | nc $IP 22
# If banner shows but ssh hangs, it's a KEX/cipher negotiation issue

Fix: explicit cipher and KEX selection

ssh \
  -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null \
  -o ConnectTimeout=10 \
  -o KexAlgorithms=curve25519-sha256 \
  -o HostKeyAlgorithms=ssh-ed25519 \
  -c aes128-ctr \
  -m hmac-sha2-256 \
  root@$IP 'echo OK'

These options force a minimal, widely-supported algorithm set that avoids negotiation hangs. Store this as a shell alias or function when working with a problematic host.

Alternative: SSH config snippet for problematic hosts

Host problematic-vps
    HostName 38.49.217.154
    KexAlgorithms curve25519-sha256
    HostKeyAlgorithms ssh-ed25519
    Ciphers aes128-ctr
    MACs hmac-sha2-256

Password auth with sshpass

When key auth isn't set up yet (fresh VPS, only root password available):

# Install if needed
sudo apt-get install -y sshpass

# Use with single quotes to protect special characters
sshpass -p 'password' ssh <user>@<host> 'command'

# For passwords with tricky characters, use a temp file
printf '%s' 'password' > /tmp/pass.txt
sshpass -f /tmp/pass.txt ssh <user>@<host> 'command'
rm /tmp/pass.txt

Auth debugging

If password is rejected, check what methods the server accepts:

ssh -v -o PreferredAuthentications=publickey,password <user>@<host> 2>&1 | \
  grep -E 'Authentications that can continue|permission|denied|method'

Key lines:

• Authentications that can continue: publickey,password — password auth is enabled
• Authentications that can continue: publickey — password is disabled, need key
• we sent a password packet, wait for reply then Permission denied — wrong password

Password auth troubleshooting

• Ubuntu 24.04 cloud images often lock the root account — password auth works on console but not over SSH.
• If sshpass with correct password returns "Permission denied," check PermitRootLogin and PasswordAuthentication in /etc/ssh/sshd_config.
• Use ssh -v and grep for Authentications that can continue to see what methods the server offers.
• Backtick gotcha: Backtick characters (`) in chat messages are often markdown formatting, not part of the actual password. If a password is shared like the password is `abc`123 — test with AND without the backticks.
• fail2ban: Don't loop on wrong password — after 2-3 failed attempts, stop and ask. Many hosts have fail2ban that will block your IP.

sshd vs ssh service name (Ubuntu 24.04)

On Ubuntu 24.04, the SSH service is named ssh, not sshd. If systemctl reload sshd fails with "Unit sshd.service not found", use systemctl reload ssh instead.

Output-size hang with custom cipher config

On servers that require explicit KEX/cipher flags (see "SSH connection troubleshooting" above), command output exceeding approximately 800 bytes hangs indefinitely. This affects cat, head, sed, and Python printing — any command producing more than a trivial amount of output.

Symptom: wc -l <file> works (output is small), but cat <file> or head -50 <file> hangs/timeouts on the same file. whoami, echo OK, and other tiny-output commands work fine. The cutoff appears around 800–1000 bytes.

Workarounds:

1. Check file size first: wc -c <file> — if under ~700 bytes, safe to cat directly.
2. Read in small chunks: sed -n '1,10p' <file>, sed -n '11,20p' <file>, etc. Keep each chunk under 700 bytes.
3. Use commands with inherently small output: xxd <file> | head -5, file <file>, wc -l <file>.
4. For configuration files, prefer grep for specific keys rather than reading the whole file.
5. For long transcripts/doctor output, grep for status lines: gbrain doctor 2>&1 | grep -E '\[(OK|WARN|FAIL)\]'

Root cause: the aes128-ctr cipher + explicit KEX settings create a packet-fragmentation issue when the SSH channel sends more than ~800 bytes in a single burst. This is NOT a general SSH issue — it only manifests on hosts that need the explicit cipher workaround. Normal SSH connections without custom cipher config don't have this problem.

File transfer (scp) with custom cipher hosts

When scp is needed to transfer files to a host that requires explicit cipher/KEX flags, use -o for all options — the scp command does NOT support the -m flag (unlike ssh):

scp -o KexAlgorithms=curve25519-sha256 \
    -o HostKeyAlgorithms=ssh-ed25519 \
    -c aes128-ctr \
    -o MACs=hmac-sha2-256 \
    local-file vps:/remote/path

Note: -o MACs= (not -m), -c for cipher, and all other flags via -o.

Tailscale Setup

curl -fsSL https://tailscale.com/install.sh | sh
tailscale up

tailscale up prints a one-time auth URL and hangs waiting for browser authentication. Run it as a background process while the user authenticates in their browser.

Auth URL behavior

• The URL changes every time tailscale up is restarted — old URLs expire immediately
• If the user doesn't authenticate in time, kill the process and re-run tailscale up to get a fresh URL
• Running tailscale status while tailscale up is waiting shows Logged out. + a potentially different URL — ignore this. Only the URL from the running tailscale up process matters
• Once authenticated, tailscale status shows the node as online with its tailnet IP
• Free tier: 100 devices, 3 users — sufficient for single-VPS builds

Pitfall

Multiple tailscale up processes will conflict. Before restarting, kill any existing tailscale up process first.

SSH key bootstrap via VNC/console

When password auth is locked but the VPS provider offers a web console (VNC):

1. Generate an SSH key locally (if not already present):

   ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ""
   

2. Give the operator your public key:

   cat ~/.ssh/id_ed25519.pub
   

3. Operator runs these commands ONE AT A TIME in the VNC console (do NOT combine with && — paste each line separately):

   First, create the directory:

   mkdir -p /root/.ssh
   

   Then write the key. Use double quotes (") around the key — single quotes can cause the file to be empty on some VNC consoles:

   echo "ssh-ed25519 AAAAC3NzaC1l... user@host" > /root/.ssh/authorized_keys
   

   Then verify the key was written:

   cat /root/.ssh/authorized_keys
   

4. Fix permissions (CRITICAL — SSH silently rejects keys with wrong perms):

   chmod 700 /root/.ssh
   chmod 600 /root/.ssh/authorized_keys
   

Pitfall — bare paste: If the operator pastes the key without the echo "..." wrapper, the shell will try to execute it as a command and fail with "ssh-ed... command not found." Always provide the full echo "..." line.

Pitfall — quoting: Use double quotes (echo "key"), not single quotes (echo 'key'). On some VNC/web consoles, single-quoted strings get mangled during copy-paste and produce empty files. If cat /root/.ssh/authorized_keys shows nothing, re-run with double quotes.

Pitfall — > vs >>: Use > (overwrite) when setting up the key for the first time — this is idempotent on re-run. Only use >> (append) if you are intentionally adding a second key to an existing file. Using >> on repeated failed attempts duplicates the key line.

Pitfall — permissions: Even with the key correctly written, SSH will reject it if /root/.ssh is not mode 700 or authorized_keys is not mode 600. Always run the chmod commands and verify.

Provisioning checklist pattern

When given a structured provisioning plan (Phase 0, Phase 1, etc.):

• Respect phase boundaries — don't start Phase N+1 work even if it "seems obvious"
• Track progress in a status file (e.g., /var/log/hermes/phase-0-status.json) so re-runs pick up where they left off
• Each step should be idempotent: check if work is already done before doing it
• Log all commands to /var/log/hermes/actions.log with timestamp, command, exit code, and rationale

Resumption audit (when continuing a partially-done provision)

When resuming a provision started in a prior session, do NOT trust the status file alone — prior runs may have left it sparse or empty. Run a bulk audit first: one SSH command that checks all key indicators at once (OS, hostname, users + groups, SSH config, UFW status, Tailscale status, Docker version + compose + daemon.json existence, /data/ tree, /srv/stack/, restic, cron entries, actions log). This 5–10 second check gives a definitive picture of what's done vs pending without guessing from an unreliable status file.

SSH verification: test from the agent's machine, not localhost

When verifying non-root SSH access (Phase 0.2 / 0.9), do NOT test with sudo -u <user> ssh localhost from the VPS — this triggers localhost key prompts and may fail even when everything is correct. The real verification is an SSH connection from the agent's machine (WSL, your workstation) directly to the VPS as the non-root user. If the VPS requires explicit KEX/cipher flags (see SSH troubleshooting above), apply those flags to the test command too.

Typical Phase 0 tasks (Ubuntu 24.04)

1. System baseline: OS confirm, apt update/upgrade, hostname, timezone UTC, unattended-upgrades (security only)
2. User and SSH: non-root sudo user, key-only SSH, docker group
3. Firewall: UFW (deny incoming, allow outgoing, allow SSH 22), Tailscale
4. Docker: Engine + Compose v2 from official repo, journald log driver, ulimit config. Critical: Docker installs often complete without /etc/docker/daemon.json — always check for it and create it explicitly. Use templates/docker-daemon.json as the starting point. After writing, run systemctl reload docker. Pitfall — host access to container databases: When exposing a Dockerized Postgres on a host port (127.0.0.1:5432:5432), connections from the host to the container arrive from the Docker bridge IP (typically 172.x.0.1), not from 127.0.0.1. If pg_hba.conf only allows local loopback, connections will fail with FATAL: no pg_hba.conf entry for host "172.18.0.1". Fix: set POSTGRES_HOST_AUTH_METHOD=trust in the container environment for local-only setups (the container is already bound to 127.0.0.1 on the host side, so remote hosts cannot reach it). Pitfall — PostgreSQL config quoting: In postgresql.conf, string values use single quotes, never double quotes. listen_addresses = "*" (double quotes) causes a syntax error and postgres won't start. Use listen_addresses = '*' (single quotes). Double quotes in PostgreSQL config are for identifiers, not values.
5. Directory layout: /data/ tree (raw, processed, brain, postgres, redis, minio, logs), /srv/stack/
6. Backups: restic + Backblaze B2, backup script, cron. B2 is optional — the operator may decline it. If declined, skip restic initialization, mark the task as skipped in the status file, and note that pg_dump-only local backups should be added once databases exist (Phase 1+).
7. Health monitoring: healthcheck.sh (disk, memory, load, containers), cron. See templates/healthcheck.sh for a working template. Pitfall: avoid bc for floating-point comparisons in bash healthcheck scripts — bc -l in subshells can hang silently on some systems. Use integer math (cut -d. -f1 for load, integer division for percentages) instead.
8. Self-config: agent state directory, phase tracking, secrets
9. Final verification: run all checks, report summary

See references/phase-0-template.md for the full detailed task list. See references/gbrain-install.md for gbrain setup (Phase 1 knowledge-brain stack).