Homelab Bootstrap

Homelab Bootstrap — Known Issues & Fixes

Hard-won lessons from bootstrapping the homelab-platform terraform-runner and provisioning VMs via Terraform + Ansible.

Ansible Installation

Never use apt install ansible — it installs a version missing the Python libraries needed by community.docker modules.

Correct install (on the VM):

curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc

uv tool install ansible-core --with ansible --with docker --with requests
export PATH="$HOME/.local/bin:$PATH"

ansible-galaxy collection install community.docker community.general

Adding Python packages to an existing ansible-core uv install:

# DON'T: uv tool run --from ansible-core pip install <pkg>  (pip not available)
# DO:
uv tool install ansible-core --with <package> --force

Ansible Stdout Callback

ANSIBLE_STDOUT_CALLBACK: yaml was removed in community.general v12.

Fix: Use the built-in default callback instead:

env:
  ANSIBLE_STDOUT_CALLBACK: default
  ANSIBLE_CALLBACK_RESULT_FORMAT: yaml

Docker Group Membership

After the Docker phase of infra-runner.yml adds the user to the docker group, the current SSH session does not pick up the new group. Running container tasks in the same session will fail with permission errors.

Fix: Log out and back in, then re-run with --skip-tags docker:

exit
ssh ubuntu@<VM_IP>
ansible-playbook ... --skip-tags docker

bpg/proxmox Provider — Disk Resize During Clone

The bpg/proxmox Terraform provider fails when trying to resize a disk during the clone operation. Two failure modes:

1. "the server did not include a data object in the response" — race condition
2. "requested size (30G) is lower than current size (31.5G)" — attempted shrink

Root cause: The Ubuntu 24.04 cloud-init template created with qm resize 9000 scsi0 +28G results in a 31.5GB disk (not 30GB — base image is ~3.5GB).

Fix in terraform/modules/proxmox-vm/main.tf:

• Remove the disk block from the VM resource entirely
• Add lifecycle { ignore_changes = [disk] }
• Use a terraform_data resource with local-exec curl to call the Proxmox resize API after VM creation
• cloud-init growpart handles partition expansion on first boot automatically

lifecycle {
  ignore_changes = [disk]
}

resource "terraform_data" "disk_resize" {
  triggers_replace = { vm_id = ..., disk_gb = var.disk_gb }
  provisioner "local-exec" {
    command = <<-EOT
      curl -k -s -f -X PUT \
        -H "Authorization: PVEAPIToken=$PROXMOX_VE_API_TOKEN" \
        -d "disk=scsi0&size=${var.disk_gb}G" \
        "$PROXMOX_VE_ENDPOINT/api2/json/nodes/${var.target_node}/qemu/${proxmox_virtual_environment_vm.vm.vm_id}/resize"
    EOT
  }
  depends_on = [proxmox_virtual_environment_vm.vm]
}

bpg/proxmox Provider — Cloud-Init Drive Storage

Error: storage 'local' does not support content-type 'images'

The initialization.datastore_id in the VM resource must be set to a storage that supports images. local only has snippets enabled by default.

Fix: Use local-lvm for the initialization datastore:

initialization {
  datastore_id = "local-lvm"   # NOT "local"
  ...
}

Note: The snippet file (user-data) is still uploaded to local — that's correct and separate from the cloud-init drive datastore.

GitHub Actions Branch Protection Bypass

User-level bypass actors in GitHub rulesets do not work on personal repos — only on organization repos. Setting bypass_actors with your user ID will appear to succeed via API but current_user_can_bypass will remain "never".

Workaround: Remove the pull_request rule from the ruleset entirely for personal repos where you're the sole contributor. Keep deletion and non_fast_forward rules for safety.

GitHub Actions Runner — Ansible on Windows

Ansible does not run natively on Windows (os.get_blocking() is Linux-only). The terraform-runner bootstrap must be run on the VM itself (SSH in and run locally), not from a Windows dev machine.

ansible-playbook -i "localhost," ansible/infra-runner.yml \
  -e "ansible_connection=local" \
  ...

Infisical Secrets in Terraform — Read TF_VAR_* from Env Directly

When a Terraform provider supports env-var fallback (e.g. TECHNITIUM_HOST, TECHNITIUM_TOKEN for kenske/technitium), wire it up with an empty provider block and let the provider read process env directly. The Infisical secrets-action already exports the env vars job-wide.

# providers.tf
provider "technitium" {}   # reads TECHNITIUM_HOST / TECHNITIUM_TOKEN from env

Don't declare TF variables + pass them through ${{ env.X }} in step env: blocks:

# BROKEN — silently loses the value
- name: Terraform Apply
  env:
    TF_VAR_technitium_host: ${{ env.TECHNITIUM_HOST }}   # ← evaluates wrong
  run: terraform apply ...

Symptom: Go provider errors like Get "***/api/...": unsupported protocol scheme "" even though the secret in Infisical is correct.

Why: ${{ env.X }} in a step's env: block does not reliably pick up vars set via $GITHUB_ENV by a prior Node.js action (which Infisical's secrets-action is). GitHub still masks the value as *** in logs, so it looks set when it isn't.

Diagnosing without exposing the value: add a step that prints the length and prefix check only — never the value:

echo "TECHNITIUM_HOST length: ${#TECHNITIUM_HOST}"
if [ "${TECHNITIUM_HOST#http://}" != "$TECHNITIUM_HOST" ]; then
  echo "starts with http://: YES"
fi

Infisical OIDC — Trust-Bound to refs/heads/main Only

Homelab repo machine identities (homelab-services, homelab-platform) have their OIDC trust template scoped to sub: repo:OWNER/REPO:ref:refs/heads/main. Workflows dispatched on any other ref fail at the secrets-action step with:

##[error]Access denied: OIDC subject not allowed.
statusCode: 403, ForbiddenError

Implication for debugging: diagnostic commits that need to load Infisical secrets must land on main. Use a small PR + squash-merge instead of gh workflow run --ref <debug-branch>.

To relax (if branch builds become a frequent need): Infisical UI → Machine Identities → <identity> → OIDC Auth → broaden the sub template to repo:OWNER/REPO:ref:refs/heads/*. Weakens posture; only do this if the friction outweighs the security trade-off.

When the 403 appears, don't waste time checking secret values or path permissions — the binding is the issue.

rsync -a Won't Create Missing Parent Dirs

rsync -a fails on the destination if the parent of the target path doesn't exist. On a freshly provisioned VM, ~/services/ doesn't exist, so rsync -az ./svc/ ubuntu@vm:~/services/svc/ errors with:

rsync: [Receiver] mkdir "/home/ubuntu/services/svc" failed:
  No such file or directory (2)
rsync error: error in file IO (code 11)

Fix: create the destination directory tree over SSH before rsync, not after:

- name: Create service directories on VM
  run: |
    ssh -i "$HOME/.ssh/deploy_key" -o StrictHostKeyChecking=no \
      ubuntu@${{ steps.vm-ip.outputs.ip }} \
      "mkdir -p ~/services/svc/data"

- name: Sync service files to VM
  run: |
    rsync -az -e "ssh -i $HOME/.ssh/deploy_key -o StrictHostKeyChecking=no" \
      services/svc/ ubuntu@${{ steps.vm-ip.outputs.ip }}:~/services/svc/

Alternative: rsync --mkpath on rsync ≥ 3.2.3, but the explicit mkdir is clearer and works everywhere.

This bit deploy-dns.yml (first prod-zone deploy) and the same ordering bug exists in deploy-observability.yml — only hidden because that VM was provisioned before deploy order mattered.

Self-Hosted Runner Offline Recovery

When a workflow sits in queued for minutes against [self-hosted, *], the runner is almost certainly offline. Check from any machine with gh:

gh api repos/OWNER/REPO/actions/runners \
  --jq '.runners[] | {name, status, busy, labels: [.labels[].name]}'

If status is "offline":

1. Verify the runner VM is powered on in Proxmox.
2. SSH in. The runner is a systemd unit named actions.runner.<owner>-<repo>.<runner-name>.service. Find the exact name and restart:

   sudo systemctl list-units 'actions.runner.*' --all
   sudo systemctl status actions.runner.OWNER-REPO.RUNNER.service
   sudo systemctl start  actions.runner.OWNER-REPO.RUNNER.service
   

3. The queued job picks up automatically once the runner reports status: online — no re-dispatch needed.

The terraform-runner serves multiple repos via separate actions-runner-* directories (one per repo registration); each registration is its own systemd unit.

kenske/technitium Provider — Known Gaps

The provider (~> 0.2.2) covers zones, records, and DHCP. It does not model:

• Forwarders / recursion settings — manage via UI
• Block lists — manage via UI

Both change rarely so this is acceptable. If you ever need to manage them in code, the escape hatch is a null_resource + local-exec curl against the Technitium HTTP API endpoints (/api/settings/set, /api/settings/setBlockListUrls). See homelab-platform/docs/decisions/dns-management.md for full rationale.

Bootstrap is one-time and manual — Terraform cannot mint its own Technitium API token. Create a non-admin terraform user in the UI, grant View + Modify on Zones, generate a token via Sessions, then store TECHNITIUM_HOST + TECHNITIUM_TOKEN in Infisical at /technitium/. Full runbook: homelab-services/services/dns/BOOTSTRAP.md.

Zone-already-exists error: if a manually-created zone matches a name Terraform wants to create, either (a) delete the zone in the UI and let Terraform create it fresh, or (b) add an import block to adopt it. The former is simpler unless you have records you want to preserve.

DNS-on-VM-Create — Reusable Pattern

The homelab-platform/terraform/modules/dns-record/ module wraps a single technitium_dns_zone_record for composition next to proxmox-vm. The reusable provision-vm.yml workflow accepts an opt-in input create_dns_record: true that loads /technitium/ from Infisical alongside the standard paths.

Per-consumer setup when adding DNS to a new repo:

1. In the repo's Terraform root: declare kenske/technitium provider, add provider "technitium" {} (env-direct — see Infisical/Terraform section above), compose module "dns" { source = ".../dns-record" }.
2. In the provision workflow: with: create_dns_record: true.
3. In Infisical: grant the consumer's machine identity read on the /technitium/ path (one-time).
4. The lan zone itself is singleton and owned by homelab-services/services/dns/terraform-config/ — don't create zones in consumer repos.

Full copy-pasteable example: homelab-platform/docs/examples/new-vm-with-dns/.

Ansible 2.16+ — ANSIBLE_SSH_AGENT=auto Required

When passing ansible_ssh_private_key_file (typical for GitHub-Actions-driven Ansible runs that load the key from Infisical), Ansible 2.16+ tries to add the key to an SSH agent. If no agent is running, the task fails immediately:

[ERROR]: Task failed: Failed to authenticate: Failed to add configured
private key into ssh-agent: Cannot utilize private_key with SSH_AGENT disabled

Fix: Set ANSIBLE_SSH_AGENT: "auto" in the step's env: block so Ansible spawns an agent on demand. Required for every remote-Ansible step.

- name: Run playbook against the new VM
  env:
    ANSIBLE_HOST_KEY_CHECKING: "False"
    ANSIBLE_SSH_AGENT: "auto"
  run: ansible-playbook -i "${{ needs.provision.outputs.vm_ip }}," ...

Don't Install a Runner on Every VM (or Every Physical Box)

Default pattern for ANY host whose workflows don't need anything host-local: drive it via Ansible-over-SSH from terraform-runner, do NOT install a GitHub Actions runner on it. Same logic applies whether the host is a Proxmox VM or a physical machine.

Per-host runners cost you:

• A GH_PAT (the default GITHUB_TOKEN can't mint registration tokens)
• A separate register-runner job that installs the runner service
• PATH issues (ansible-playbook: command not found — uv tool install only adds the bin dir to PATH via ~/.bashrc, which non-interactive runner shells don't source)
• A reinstall on every reprovision
• A chicken-and-egg in provision-X.yml when the workflow that provisions the host is itself runs-on: self-hosted-<that-host> — can't bootstrap a host whose runner doesn't exist yet

Use a per-host runner only when everything host-local matters: heavy GPU access required by every step, large local-disk operations that can't tolerate SSH bandwidth, etc. Even GPU + GGUF management is fine over SSH (proven by the inference-host refactor — see /homelab-local-inference "Topology" section).

Decommissioning an orphan runner registered on a VM/box: delete from GitHub side via API (gh api -X DELETE /repos/.../actions/runners/{id}). The local systemd unit + actions-runner directory linger and are harmless clutter. Clean them up by SSH-ing in and rm -rf ~/actions-runner if the runner ran as a regular user; sudo + systemd disable/remove for the unit file. If the host is going to be reprovisioned soon, leave the local files — Terraform destroy+rebuild wipes them (VMs only).

Reference workflows in local-inference-infrastructure/.github/workflows/:

• provision-litellm-gateway.yml — gateway VM (Terraform + Ansible-SSH)
• provision-inference.yml — physical inference box (Ansible-SSH)
• sync-models.yml, deploy-alloy.yml — recurring per-host ops over SSH

All five share the same shape: runs-on: [self-hosted, self-hosted-infra], load /ansible from Infisical, write deploy key, run ansible-playbook -i "$IP," ..., clean up.

DNS Resolution From terraform-runner — dig @<dns-vm> at Workflow Time

terraform-runner sits on 192.168.1.0/24; the authoritative .lan resolver (dns-vm) is on 192.168.0.250. terraform-runner does NOT use dns-vm as its system resolver, so .lan hostnames don't resolve through the normal stack:

ssh: Could not resolve hostname ubuntu-tower.lan: Name or service not known

This bit the inference-host refactor on first try. Two options:

Don't: reconfigure terraform-runner's /etc/systemd/resolved.conf to use dns-vm. Works but you're now silently dependent on dns-vm for every DNS lookup the runner makes (apt, docker pulls, GitHub API, etc.).

Do: have the workflow resolve .lan names against dns-vm explicitly, just before the ansible step:

- name: Resolve <hostname> to IP via dns-vm
  id: target
  run: |
    IP=$(dig @192.168.0.250 +short "$HOSTNAME" A 2>/dev/null | head -n1)
    if [ -z "$IP" ]; then
      IP=$(nslookup "$HOSTNAME" 192.168.0.250 2>/dev/null | awk '/^Address: 192\./ {print $2; exit}')
    fi
    [ -n "$IP" ] || { echo "ERROR: $HOSTNAME did not resolve"; exit 1; }
    echo "ip=$IP" >> "$GITHUB_OUTPUT"

- name: Run ansible
  run: |
    ansible-playbook -i "${{ steps.target.outputs.ip }}," ...

DHCP renumbering becomes transparent — every run gets a fresh lookup. dig is preferred; nslookup fallback covers the case where bind9-dnsutils isn't installed on the runner.

The manifest / inventory keeps .lan hostnames as the source of truth. Only the SSH target changes (resolved IP instead of FQDN).

Ad-Hoc Ansible Inventory -i "<host>," Requires hosts: all

Common pattern in CI: pass the target via -i "<ip-or-host>," (the trailing comma is what tells ansible "this is an inline inventory, not a file"). The single host lands in the all and ungrouped groups — NOT in any named group.

If the playbook header is:

- name: My playbook
  hosts: inference_machines

…it'll silently no-op the entire run with:

[WARNING]: Could not match supplied host pattern, ignoring: inference_machines
PLAY RECAP *********
(no PLAY RECAP rows, 0 hosts matched)

And the job will report "success" because zero-hosts-matched isn't a failure to ansible. Real bug, silent symptom.

Fix: use hosts: all in playbooks that are invoked via inline inventory:

- name: My playbook
  hosts: all   # works with -i "host," AND with inventory.ini group filters

Local-debug runs with the inventory file still work because all implicitly includes every host in the inventory.

gather_facts: true When You Use ansible_env.*

When a playbook references {{ ansible_env.HOME }} or any other fact (ansible_user_id, ansible_distribution, etc.), facts must be gathered first. If you disabled fact-gathering for speed:

- hosts: all
  gather_facts: false   # ← culprit
  tasks:
    - stat:
        path: "{{ ansible_env.HOME }}/.lmstudio/bin/lms"   # FAILS

…every task that touches ansible_env fails with:

Finalization of task args failed: Error while resolving value for 'path':
'ansible_env' is undefined

Fix: either set gather_facts: true (the default — just delete the override), or pass the value via extra-vars instead of a fact:

gather_facts: true
# …or:
# -e remote_home=/home/blake

Make Provisioning Playbooks Self-Skip on Already-Done Phases

For playbooks that should be safe to re-run on already-provisioned hosts AND to provision fresh ones, gate each disruptive phase on a cheap precheck rather than relying on each task's own idempotency. Two patterns from inference-setup.yml:

Skip disk-expand on physical boxes without cloud-image LVM:

- name: "[disk] Precheck: does /dev/ubuntu-vg/ubuntu-lv exist?"
  stat:
    path: /dev/ubuntu-vg/ubuntu-lv
  register: lv_stat
  tags: disk

- name: "[disk] Extend LV"
  command: lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
  when: lv_stat.stat.exists
  tags: disk
# …same when: on resize2fs

A bare-metal install without LVM gets a "skipped" message instead of an lvextend failure.

Skip NVIDIA driver install + reboot when driver already works:

- name: "[drivers] Precheck: nvidia-smi works?"
  command: nvidia-smi
  register: nvidia_smi_precheck
  ignore_errors: true
  changed_when: false
  tags: drivers

- name: "[drivers] Install ubuntu-drivers-common"
  apt: { name: ubuntu-drivers-common, state: present, update_cache: true }
  become: true
  when: nvidia_smi_precheck.rc != 0
  tags: drivers
# …same when: on each driver-install and reboot task

A box that already has a working driver gets a fast no-op; a fresh box installs, reboots, and the playbook intentionally fails with a "REBOOT IN PROGRESS — re-run with skip_tags=disk,drivers" message so the workflow operator knows to dispatch a resume.

The cost is ~3 added tasks per phase. The benefit is one playbook for fresh-vs-existing and confidence to re-run after every commit.

Best-Effort Boot-Time Side Effects

Systemd ExecStartPost and ansible tasks that do "warm-start" optimizations (preloading a model, warming a cache) should be tolerant of missing prerequisites — otherwise they block fresh-box provisioning on something that's downloaded later by a different workflow.

Systemd: prefix the line with - to ignore exit codes:

[Service]
ExecStartPost=/path/lms server start ...
-ExecStartPost=/path/lms load qwen3.5-9b -c 32768 -y   ← `-` makes it best-effort

Ansible: failed_when: false, register the result, optionally emit a debug note:

- name: "[lmstudio] Preload reasoning model (best-effort)"
  shell: "{{ ansible_env.HOME }}/.lmstudio/bin/lms load qwen3.5-9b -c 32768 -y"
  register: lms_load_result
  changed_when: lms_load_result.rc == 0
  failed_when: false

- name: "[lmstudio] Note if preload was skipped"
  debug:
    msg: "{{ 'Preload OK' if lms_load_result.rc == 0 else 'Skipped (model not downloaded yet — sync-models will fetch).' }}"

The truthful version of "skipped" beats a red-X job for a step the operator never intended to be a gate.

One-Time Deploy-Key Bootstrap for New Ansible-SSH Hosts

When adding a new host to the Ansible-over-SSH cluster (gateway, inference, whatever), the Ansible deploy public key needs to land in the target user's ~/.ssh/authorized_keys BEFORE any workflow can SSH in. Pattern from local-inference-infrastructure/scripts/bootstrap-deploy-key.sh:

1. Pull ANSIBLE_PRIVATE_KEY from Infisical (/ansible, prod) via infisical secrets get ANSIBLE_PRIVATE_KEY --plain into mktemp(1).
2. Derive the public key locally: ssh-keygen -y -f /tmp/key > /tmp/key.pub.
3. ssh-copy-id -i /tmp/key.pub user@new-host.lan (prompts for password once; idempotent — a re-run skips if the key already works).
4. trap rm -f /tmp/key /tmp/key.pub EXIT so nothing leaks even on Ctrl-C.

Reusable, idempotent, no secrets persisted to disk outside /tmp. Works for any host whose <user> has password auth enabled at least once.

Don't try to drive this from a workflow — the workflow ITSELF needs the key to be in place. It's a strictly local one-time op (a human runs the script from a shell where they're logged into Infisical).

git add -A in User-Shared Working Trees Sweeps WIP

When committing inside a repo the user actively works in (i.e. almost any homelab repo), use git add -- <specific-paths> rather than git add -A. The user often has uncommitted WIP in the working tree (in-progress playbooks, vars, etc.); git add -A pulls those into a commit whose message doesn't describe them, and the no-PR direct-push policy on these repos means there's no review gate to catch it.

Habit:

git status --short                    # confirm what's modified
git add -- path/I/edited.yml ...      # explicit
git status --short                    # confirm staged set matches intent
git commit -m "..."

git add -A is fine in fresh worktrees or repos with no human collaborator in parallel.

Per-Repo Runner Registration

terraform-runner (label self-hosted-infra) needs to be registered separately for every repo it serves. Registration is per-repo, not per-org for personal accounts.

When you add a new repo whose workflows target [self-hosted, self-hosted-infra], dispatch homelab-platform/.github/workflows/register-runner.yml against the new repo URL — otherwise jobs queue forever. The runner can be registered to N repos simultaneously; each gets its own actions-runner-<repo> directory and its own systemd unit.

Infisical-action — Pin to v1.0.15 (last Node 20)

Infisical/secrets-action@v1.0.16 was retagged April 2026 to switch from Node 20 to Node 24. Older self-hosted runner binaries (2.322.0 and below) don't support Node 24:

##[error]Can't use 'using: node24' as it's not supported in the runtime

Fix: Pin to @v1.0.15 in every workflow on every repo until the self-hosted runner binary is upgraded.

- uses: Infisical/secrets-action@v1.0.15

Long-term fix: Upgrade actions-runner to a version that ships Node 24 (2.324+), then unpin.

Multi-Project Infisical Pattern

Default homelab setup uses two Infisical projects per repo that needs secrets:

• Shared infra project (homelab-platform-eu2-m): holds /proxmox, /terraform, /ansible — used by every repo that calls the reusable provision-vm.yml workflow. Referenced via vars.INFISICAL_PROJECT_SLUG.
• Per-repo project (e.g. local-inference-infrastructure-pca-l): holds app-scoped secrets like /litellm. Referenced via a repo-specific variable like vars.LITELLM_INFISICAL_PROJECT_SLUG.

The same OIDC machine identity (vars.INFISICAL_IDENTITY_ID) is bound to both projects, so only the project-slug differs between secret-load steps. Trust binding on the identity must include both projects — granted in the Infisical UI under the identity's "Project Access" tab.

# Shared infra path
- uses: Infisical/secrets-action@v1.0.15
  with:
    identity-id:  ${{ vars.INFISICAL_IDENTITY_ID }}
    project-slug: ${{ vars.INFISICAL_PROJECT_SLUG }}
    secret-path:  /ansible

# Per-repo path
- uses: Infisical/secrets-action@v1.0.15
  with:
    identity-id:  ${{ vars.INFISICAL_IDENTITY_ID }}
    project-slug: ${{ vars.LITELLM_INFISICAL_PROJECT_SLUG }}
    secret-path:  /litellm

Verifying Loaded Secrets Without Leaking Them

When debugging "extra-var came through empty," don't echo the value. Echo the length:

for v in LITELLM_MASTER_KEY LITELLM_POSTGRES_PASSWORD; do
  len=$(eval echo \${#$v})
  [ "$len" -eq 0 ] && echo "MISSING: $v" || echo "OK: $v ($len chars)"
done

Zero-length means the secret-path doesn't contain that key, or the key exists in a different env / project. Non-zero confirms the action loaded it without exposing the value in logs. Pair with the length-only debug pattern in the Infisical/Terraform section above.

gh run rerun --failed Uses the Original SHA's Workflow File

Counterintuitive: re-running a failed job does not pick up workflow edits made after the original run started. The workflow YAML is pinned to the head SHA of the original dispatch.

When testing a workflow fix, always do a fresh gh workflow run after the commit; don't --failed-rerun.

infra-runner.yml — svc.sh Guard When Re-Registering

The runner tag in homelab-platform/ansible/infra-runner.yml historically ran ./svc.sh stop and ./svc.sh uninstall unconditionally. On a fresh runner directory those scripts don't exist yet, and the task fails:

/bin/sh: 1: ./svc.sh: not found  (exit 127)

failed_when didn't catch "not found" — only "not installed". The fix (now upstream in homelab-platform) is a stat check before stop/uninstall:

- name: "[runner] Check whether svc.sh exists"
  stat:
    path: "{{ github_actions_runner_dir }}/svc.sh"
  register: runner_svc_script
  tags: runner

- name: "[runner] Stop existing service if present"
  shell: ./svc.sh stop
  args: { chdir: "{{ github_actions_runner_dir }}" }
  become: true
  when: runner_svc_script.stat.exists
  ...

Any repo that forks or copies infra-runner.yml needs the same guard.

Bind-Mounted Config Files That Don't Exist Yet

Pattern in litellm-gateway-setup.yml: the LiteLLM container is started with command: --config /app/litellm/litellm-config.yaml, where the config file is bind-mounted from ~/litellm/. But the config is generated and shipped by a separate workflow (deploy-litellm.yml) on every push.

On a brand-new VM, the provision playbook leaves the container in a crash-loop until deploy-litellm runs at least once. This is expected behavior — the first deploy-litellm after provisioning fixes it.

When adding a similar pattern: document the two-step bring-up explicitly, or have the provision workflow trigger the deploy workflow at the end via gh workflow run.

gh --jq Instead of External jq in Monitor/Background Bash

When monitoring GitHub Actions runs from Claude Code on Windows via the Monitor tool, the background bash environment does not have jq on PATH. Loops that pipe through jq silently produce empty output, and events never fire.

Fix: Use gh's built-in --jq flag instead of an external jq:

gh run view $ID --json status,conclusion,jobs \
  --jq '.jobs[] | "\(.name): \(.status)/\(.conclusion)"'

Avoid ... | jq in any script you intend to run inside the Monitor tool or as a background Bash on a Windows host.

dns-record Module — Prefer CNAME for VM Aliases

When using the dns-record module from homelab-platform to create a friendly alias for a VM (e.g. dashboard.lan for dashboard-vm), use the CNAME mode pointing at the VM's auto-registered hostname, not the A-record mode pinned at module.vm.vm_ip:

module "dns" {
  source   = "github.com/BlakeHastings/homelab-platform//terraform/modules/dns-record?ref=main"
  hostname = "dashboard"                       # → dashboard.lan
  cname    = "${module.vm.vm_name}.lan"        # → dashboard-vm.lan
  zone     = "lan"
}

Why: The router auto-registers each VM's hostname into Technitium as a TTL-900 A record on DHCP lease (visible as <vm_name>.lan in records lan). When the VM's IP changes (DHCP renewal without a reservation), the router pushes the new A record automatically. A CNAME pointing at <vm_name>.lan follows for free; a Terraform-managed A record pinned at module.vm.vm_ip does not, and goes stale until terraform apply runs again.

Symptom of the wrong choice: dashboard.lan returns the wrong IP after a DHCP renewal while dashboard-vm.lan resolves correctly. Fix is to switch from ip = to cname = and re-provision.

When the A-record mode IS correct:

• DHCP reservation on the router pins the MAC → IP (record stays valid)
• Target is not a VM (Proxmox host, external service) — no router auto-registration to chain off (use terraform-config/a_records map)
• Static IP assignment

Full module docs: homelab-platform/terraform/modules/dns-record/README.md.

GitHub Actions — Zombie "In-Progress" Runs

Distinct from the runner-offline failure mode above. Symptom:

• Runner status shows online, busy: true
• Specific run shows in_progress for many minutes
• Step-level gh run view <id> --json jobs shows every step has status: completed, conclusion: success — yet the run-level state remains in_progress
• New queued runs cannot start (runner is "busy" with the zombie)

This is a stale-state bug where the runner's "I finished" heartbeat never reached GitHub (network blip, runner restart, OOM kill of the runner process mid-cleanup, etc.). The actual work is done; only the control-plane status is wrong.

Recovery:

# 1. Confirm steps actually completed
gh run view <ID> --json jobs --jq '.jobs[0].steps[] | {name, status, conclusion}'

# 2. Cancel the zombie (this often won't propagate to the runner,
#    but flips the API state so queued runs can proceed)
gh run cancel <ID>

# 3. SSH into the runner VM and restart the systemd unit (same as
#    the offline-recovery pattern above)
sudo systemctl restart actions.runner.OWNER-REPO.RUNNER.service

After step 3, the zombie flips to failure/cancelled and queued runs start within seconds.

Diagnostic shortcut: if all steps show completed/success but the run-level conclusion is empty/null and time-since-last-step exceeds ~2 minutes, it's a zombie. Don't wait it out — restart the runner.

Terraform — Output Description Fields Don't Allow ${} Interpolation

Subtle but recurring. Bites at plan time with a confusing message:

Error: Variables not allowed
  on main.tf line 35, in output "vm_ip":
   35:   description = "VM IP, also reachable as ${module.dns.fqdn}"
Variables may not be used here.

Output description (and variable description) fields must be literal strings — Terraform evaluates them statically for terraform output -json introspection, before any module wiring is resolved.

Fix: make the description a plain string. Reference the related value as a separate output so consumers can introspect both:

output "vm_ip" {
  description = "VM IP — also reachable by FQDN from the fqdn output below."
  value       = module.vm.vm_ip
}

output "fqdn" {
  description = "Friendly alias FQDN (a CNAME to the router-registered hostname)."
  value       = module.dns.fqdn
}

Same rule applies to:

• variable "x" { description = "..." } — no interpolation
• output "x" { description = "..." } — no interpolation
• Most resource attribute description fields (varies by provider)

Interpolation IS allowed in value, default, and most other fields.

localhost Inside a Bridge-Networking Container is the Container, Not the Host

Surprising every time. A container in default Docker bridge networking that does curl http://localhost:3100 hits the container itself, not the host's 3100. So:

• Prometheus container scraping localhost:9100 — scrapes Prometheus, not the host's Node Exporter.
• Alloy container pushing to http://localhost:3100/loki/api/v1/push — pushes to Alloy itself, not the host's Loki.

The host's bound port (-p 3100:3100) is reachable from inside the container at the host's LAN IP / hostname, not at localhost. Two right ways:

1. Hostname — <host>.lan:<port> (resolves to host's LAN IP, reaches the published Docker port). Preferred — survives DHCP and the same config works from other hosts too.
2. host.docker.internal — Docker's special hostname for the host gateway. On Linux requires extra_hosts: ["host.docker.internal:host-gateway"] in compose. Less portable.
3. Host networking — network_mode: host puts the container ON the host's network namespace. Then localhost IS the host. Node Exporter uses this pattern (it must, to read host metrics). For most services it's overkill and removes the published-port isolation.

If a stack works fine when you curl from outside the VM but fails when one container talks to another, suspect this. Especially common for Prometheus self-scrapes and Alloy pushing to a local Loki.

docker compose up -d Won't Recreate a Container After a Bind-Mount File Change

Compose's up -d is idempotent on its own model — image, env, ports, volumes-as-paths. It does not look at the content of files inside bind-mounted volumes. So if you:

1. Change a config file on the host that's bind-mounted into a container
2. Run docker compose up -d

…the container stays unchanged, still reading the old in-memory config. Same gotcha if the container CrashLooped from a prior bad config — up -d sees it as "already there, restart-policy will keep handling it" and leaves it.

Force recreation:

docker compose up -d --force-recreate

…recreates every container in the compose file even when nothing in the compose model changed. The cost is 5-30s of downtime per service. For deploys triggered by config-file edits (e.g. our deploy-observability), this is the right default — guarantees the new config is actually in effect.

Alternative for surgical recreation: docker compose up -d --force-recreate <service>.

Ansible community.docker.docker_container Idempotency Doesn't See Bind-Mount Files

Parallel of the docker-compose gotcha. docker_container checks the container's image, env, ports, volume paths — but not the contents of the files at those volume paths. So if a template: task updates ~/alloy/config.alloy (bind-mounted into the container), the subsequent docker_container task with state: started sees no change and does nothing.

Fix: handler triggered from the template task.

- name: Render config
  template:
    src: foo.j2
    dest: ~/foo/config.yaml
  notify: restart foo

- name: Run container
  community.docker.docker_container:
    name: foo
    image: foo:latest
    volumes:
      - ~/foo/config.yaml:/etc/foo/config.yaml
    # ... (idempotent on its own parameters only)

handlers:
  - name: restart foo
    community.docker.docker_container:
      name: foo
      state: started
      restart: true   # stop + start; container re-reads the bind-mount on boot

Caveat: handlers only fire when the notifying task reports changed. If the on-disk file already matches what the template would render (e.g. manual edit, or previous run already applied the new value), no diff → no notify → no restart. For pathological cases (state drift between disk and in-memory), add a manual docker restart <name> step or run the play with --force-handlers.

docker_container Is Additive: It Never Reaps Containers You Stop Declaring

community.docker.docker_container only manages the containers a playbook currently declares. Delete a container's task (or move the service to a different host) and re-running the playbook does nothing to the old container — it keeps running, orphaned, because nothing references it anymore. There is no state: present reconcile-the-whole-host semantic the way docker compose down removes services dropped from the compose file. Re-applying config is not a cleanup mechanism.

Consequence for decommissioning: "I removed it from the playbook, so re-running provisioning will clean the box" is wrong. To actually remove a service you have two choices:

• An explicit state: absent task (the durable, reproducible option; keep it in the playbook so any future re-provision is self-healing). See the legacy-Playwright cleanup block in local-inference-infrastructure/ansible/litellm-gateway-setup.yml for the pattern.
• A one-off docker rm -f <name> over SSH (fast, but not captured in config, so the orphan can reappear if anything ever re-runs the old path).

Watch for dependency stragglers: removing open-webui and litellm leaves litellm-postgres (their backing DB) running untouched, because it was its own docker_container task. Always docker ps -a after a manual removal and reconcile against what the playbook should leave behind.

systemd-resolved Drop-In for Routing One Domain to One DNS Server

Ubuntu 24.04 uses systemd-resolved by default. To route only .lan queries to Technitium (192.168.0.250) while keeping the system's default resolvers for everything else, write a drop-in:

# /etc/systemd/resolved.conf.d/technitium.conf
[Resolve]
DNS=192.168.0.250
Domains=~lan

Then sudo systemctl restart systemd-resolved. The ~lan syntax marks the domain as routing-only — systemd-resolved sends only *.lan queries to that DNS server, and other queries continue to the system's configured resolvers.

This pattern is the right answer for "give this device .lan resolution without changing its primary DNS" — used today in homelab-services/.github/workflows/deploy-observability.yml (for the runner) and centralized in homelab-platform/ansible/base-vm.yml (for every provisioned VM).

The dropin file persists across reboots and survives cloud-init re-runs (cloud-init writes /etc/netplan/..., not resolved.conf.d).

Container User UIDs for Bind-Mount Permissions

When a containerized service writes to a bind-mounted host directory, the host directory must be owned by the UID the container runs as. The process inside the container can't see host usernames — only numeric UIDs map across the namespace boundary. If perms are wrong the container CrashLoops on startup ("permission denied" writing to its data dir).

Reference table for the services we run:

Idiomatic deploy fix — in the workflow's mkdir step, also chown:

ssh ubuntu@$VM_IP <<EOSH
  mkdir -p ~/services/observability/data/{grafana,prometheus,loki,tempo}
  sudo chown -R 472:472     ~/services/observability/data/grafana
  sudo chown -R 65534:65534 ~/services/observability/data/prometheus
  sudo chown -R 10001:10001 ~/services/observability/data/loki
  sudo chown -R 10001:10001 ~/services/observability/data/tempo
EOSH

Idempotent — re-chown is a no-op once ownership is correct.

To find a new image's UID: docker run --rm --entrypoint id <image> — output line is uid=NNN(name) gid=NNN(name) ....

GitHub Actions: workflow_dispatch 422 Means the YAML Won't Parse

Symptom:

could not create workflow dispatch event: HTTP 422:
Workflow does not have 'workflow_dispatch' trigger
(https://api.github.com/repos/OWNER/REPO/actions/workflows/<id>/dispatches)

Misleading. The workflow DOES have `on: w

…(truncated)