# Homelab Bootstrap

> Reference guide for homelab-platform and homelab-services gotchas, known issues, and hard-won fixes. Use when troubleshooting Ansible (handler patterns, docker_container idempotency on bind-mount file changes, docker_container being additive so removing/decommissioning a service needs state: absent or docker rm, copy: vs template: for placeholder substitution, ANSIBLE_SSH_AGENT auto for 2.16+, ansible_hostname drift), Terraform (bpg/proxmox provider, kenske/technitium provider, dns-record CNAME vs A-record, output description interpolation, stale vm_ip after DHCP renewal), GitHub Actions (runner registration, offline / zombie-busy recovery, workflow_dispatch 422 from broken YAML, heredoc-in-block-scalar collisions), Infisical (OIDC trust bindings, env var pickup in Terraform), Docker (localhost in bridge containers, force-recreate after bind-mount edit, container UIDs for bind-mount permissions), DNS (.lan via systemd-resolved drop-ins, static-IP hosts need explicit A records), rsync deploy steps, or VM provi

- Skill: `blakehastings/homelab-bootstrap` (Agent Skill)
- Install (CLI): `npx skillmds@latest add blakehastings/homelab-bootstrap`
- Raw SKILL.md: https://api.skillmd.com/api/skills/blakehastings/homelab-bootstrap/raw
- Safety review: WARNING
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: BlakeHastings (https://skillmd.com/u/blakehastings)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/blakehastings/homelab-bootstrap

---


# Homelab Bootstrap — Known Issues & Fixes

Hard-won lessons from bootstrapping the homelab-platform terraform-runner
and provisioning VMs via Terraform + Ansible.

---

## Ansible Installation

**Never use `apt install ansible`** — it installs a version missing the Python
libraries needed by `community.docker` modules.

### Correct install (on the VM):
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc

uv tool install ansible-core --with ansible --with docker --with requests
export PATH="$HOME/.local/bin:$PATH"

ansible-galaxy collection install community.docker community.general
```

### Adding Python packages to an existing ansible-core uv install:
```bash
# DON'T: uv tool run --from ansible-core pip install <pkg>  (pip not available)
# DO:
uv tool install ansible-core --with <package> --force
```

---

## Ansible Stdout Callback

`ANSIBLE_STDOUT_CALLBACK: yaml` was removed in `community.general` v12.

**Fix:** Use the built-in default callback instead:
```yaml
env:
  ANSIBLE_STDOUT_CALLBACK: default
  ANSIBLE_CALLBACK_RESULT_FORMAT: yaml
```

---

## Docker Group Membership

After the Docker phase of `infra-runner.yml` adds the user to the `docker`
group, **the current SSH session does not pick up the new group**. Running
container tasks in the same session will fail with permission errors.

**Fix:** Log out and back in, then re-run with `--skip-tags docker`:
```bash
exit
ssh ubuntu@<VM_IP>
ansible-playbook ... --skip-tags docker
```

---

## bpg/proxmox Provider — Disk Resize During Clone

The `bpg/proxmox` Terraform provider fails when trying to resize a disk
during the clone operation. Two failure modes:

1. `"the server did not include a data object in the response"` — race condition
2. `"requested size (30G) is lower than current size (31.5G)"` — attempted shrink

**Root cause:** The Ubuntu 24.04 cloud-init template created with `qm resize 9000 scsi0 +28G`
results in a **31.5GB** disk (not 30GB — base image is ~3.5GB).

**Fix in `terraform/modules/proxmox-vm/main.tf`:**
- Remove the `disk` block from the VM resource entirely
- Add `lifecycle { ignore_changes = [disk] }`
- Use a `terraform_data` resource with `local-exec` curl to call the Proxmox
  resize API after VM creation
- `cloud-init growpart` handles partition expansion on first boot automatically

```hcl
lifecycle {
  ignore_changes = [disk]
}

resource "terraform_data" "disk_resize" {
  triggers_replace = { vm_id = ..., disk_gb = var.disk_gb }
  provisioner "local-exec" {
    command = <<-EOT
      curl -k -s -f -X PUT \
        -H "Authorization: PVEAPIToken=$PROXMOX_VE_API_TOKEN" \
        -d "disk=scsi0&size=${var.disk_gb}G" \
        "$PROXMOX_VE_ENDPOINT/api2/json/nodes/${var.target_node}/qemu/${proxmox_virtual_environment_vm.vm.vm_id}/resize"
    EOT
  }
  depends_on = [proxmox_virtual_environment_vm.vm]
}
```

---

## bpg/proxmox Provider — Cloud-Init Drive Storage

**Error:** `storage 'local' does not support content-type 'images'`

The `initialization.datastore_id` in the VM resource must be set to a storage
that supports images. `local` only has snippets enabled by default.

**Fix:** Use `local-lvm` for the initialization datastore:
```hcl
initialization {
  datastore_id = "local-lvm"   # NOT "local"
  ...
}
```

Note: The snippet file (user-data) is still uploaded to `local` — that's
correct and separate from the cloud-init drive datastore.

---

## GitHub Actions Branch Protection Bypass

User-level bypass actors in GitHub rulesets **do not work on personal repos**
— only on organization repos. Setting `bypass_actors` with your user ID will
appear to succeed via API but `current_user_can_bypass` will remain `"never"`.

**Workaround:** Remove the `pull_request` rule from the ruleset entirely for
personal repos where you're the sole contributor. Keep `deletion` and
`non_fast_forward` rules for safety.

---

## GitHub Actions Runner — Ansible on Windows

Ansible does not run natively on Windows (`os.get_blocking()` is Linux-only).
The terraform-runner bootstrap must be run **on the VM itself** (SSH in and
run locally), not from a Windows dev machine.

```bash
ansible-playbook -i "localhost," ansible/infra-runner.yml \
  -e "ansible_connection=local" \
  ...
```

---

## Infisical Secrets in Terraform — Read TF_VAR_* from Env Directly

When a Terraform provider supports env-var fallback (e.g. `TECHNITIUM_HOST`,
`TECHNITIUM_TOKEN` for `kenske/technitium`), wire it up with an **empty
provider block** and let the provider read process env directly. The
Infisical `secrets-action` already exports the env vars job-wide.

```hcl
# providers.tf
provider "technitium" {}   # reads TECHNITIUM_HOST / TECHNITIUM_TOKEN from env
```

**Don't** declare TF variables + pass them through `${{ env.X }}` in step
`env:` blocks:

```yaml
# BROKEN — silently loses the value
- name: Terraform Apply
  env:
    TF_VAR_technitium_host: ${{ env.TECHNITIUM_HOST }}   # ← evaluates wrong
  run: terraform apply ...
```

**Symptom:** Go provider errors like
`Get "***/api/...": unsupported protocol scheme ""` even though the secret
in Infisical is correct.

**Why:** `${{ env.X }}` in a step's `env:` block does not reliably pick up
vars set via `$GITHUB_ENV` by a prior Node.js action (which Infisical's
secrets-action is). GitHub still masks the value as `***` in logs, so it
looks set when it isn't.

**Diagnosing without exposing the value:** add a step that prints the
length and prefix check only — never the value:

```bash
echo "TECHNITIUM_HOST length: ${#TECHNITIUM_HOST}"
if [ "${TECHNITIUM_HOST#http://}" != "$TECHNITIUM_HOST" ]; then
  echo "starts with http://: YES"
fi
```

---

## Infisical OIDC — Trust-Bound to refs/heads/main Only

Homelab repo machine identities (homelab-services, homelab-platform) have
their OIDC trust template scoped to `sub: repo:OWNER/REPO:ref:refs/heads/main`.
Workflows dispatched on any other ref fail at the secrets-action step with:

```
##[error]Access denied: OIDC subject not allowed.
statusCode: 403, ForbiddenError
```

**Implication for debugging:** diagnostic commits that need to load Infisical
secrets must land on `main`. Use a small PR + squash-merge instead of
`gh workflow run --ref <debug-branch>`.

**To relax (if branch builds become a frequent need):** Infisical UI →
Machine Identities → `<identity>` → OIDC Auth → broaden the sub template
to `repo:OWNER/REPO:ref:refs/heads/*`. Weakens posture; only do this if
the friction outweighs the security trade-off.

When the 403 appears, don't waste time checking secret values or path
permissions — the binding is the issue.

---

## rsync -a Won't Create Missing Parent Dirs

`rsync -a` fails on the destination if the *parent* of the target path
doesn't exist. On a freshly provisioned VM, `~/services/` doesn't exist,
so `rsync -az ./svc/ ubuntu@vm:~/services/svc/` errors with:

```
rsync: [Receiver] mkdir "/home/ubuntu/services/svc" failed:
  No such file or directory (2)
rsync error: error in file IO (code 11)
```

**Fix:** create the destination directory tree over SSH **before** rsync,
not after:

```yaml
- name: Create service directories on VM
  run: |
    ssh -i "$HOME/.ssh/deploy_key" -o StrictHostKeyChecking=no \
      ubuntu@${{ steps.vm-ip.outputs.ip }} \
      "mkdir -p ~/services/svc/data"

- name: Sync service files to VM
  run: |
    rsync -az -e "ssh -i $HOME/.ssh/deploy_key -o StrictHostKeyChecking=no" \
      services/svc/ ubuntu@${{ steps.vm-ip.outputs.ip }}:~/services/svc/
```

Alternative: `rsync --mkpath` on rsync ≥ 3.2.3, but the explicit mkdir is
clearer and works everywhere.

This bit `deploy-dns.yml` (first prod-zone deploy) and the same ordering
bug exists in `deploy-observability.yml` — only hidden because that VM was
provisioned before deploy order mattered.

---

## Self-Hosted Runner Offline Recovery

When a workflow sits in `queued` for minutes against `[self-hosted, *]`,
the runner is almost certainly offline. Check from any machine with
`gh`:

```bash
gh api repos/OWNER/REPO/actions/runners \
  --jq '.runners[] | {name, status, busy, labels: [.labels[].name]}'
```

If `status` is `"offline"`:

1. Verify the runner VM is powered on in Proxmox.
2. SSH in. The runner is a systemd unit named
   `actions.runner.<owner>-<repo>.<runner-name>.service`. Find the exact
   name and restart:
   ```bash
   sudo systemctl list-units 'actions.runner.*' --all
   sudo systemctl status actions.runner.OWNER-REPO.RUNNER.service
   sudo systemctl start  actions.runner.OWNER-REPO.RUNNER.service
   ```
3. The queued job picks up automatically once the runner reports
   `status: online` — no re-dispatch needed.

The terraform-runner serves multiple repos via separate `actions-runner-*`
directories (one per repo registration); each registration is its own
systemd unit.

---

## kenske/technitium Provider — Known Gaps

The provider (`~> 0.2.2`) covers zones, records, and DHCP. It does **not**
model:

- **Forwarders / recursion settings** — manage via UI
- **Block lists** — manage via UI

Both change rarely so this is acceptable. If you ever need to manage them
in code, the escape hatch is a `null_resource` + `local-exec curl` against
the Technitium HTTP API endpoints (`/api/settings/set`,
`/api/settings/setBlockListUrls`). See
`homelab-platform/docs/decisions/dns-management.md` for full rationale.

**Bootstrap is one-time and manual** — Terraform cannot mint its own
Technitium API token. Create a non-admin `terraform` user in the UI, grant
View + Modify on Zones, generate a token via Sessions, then store
`TECHNITIUM_HOST` + `TECHNITIUM_TOKEN` in Infisical at `/technitium/`. Full
runbook: `homelab-services/services/dns/BOOTSTRAP.md`.

**Zone-already-exists error:** if a manually-created zone matches a name
Terraform wants to create, either (a) delete the zone in the UI and let
Terraform create it fresh, or (b) add an `import` block to adopt it. The
former is simpler unless you have records you want to preserve.

---

## DNS-on-VM-Create — Reusable Pattern

The `homelab-platform/terraform/modules/dns-record/` module wraps a single
`technitium_dns_zone_record` for composition next to `proxmox-vm`. The
reusable `provision-vm.yml` workflow accepts an opt-in input
`create_dns_record: true` that loads `/technitium/` from Infisical
alongside the standard paths.

Per-consumer setup when adding DNS to a new repo:

1. In the repo's Terraform root: declare `kenske/technitium` provider,
   add `provider "technitium" {}` (env-direct — see Infisical/Terraform
   section above), compose `module "dns" { source = ".../dns-record" }`.
2. In the provision workflow: `with: create_dns_record: true`.
3. In Infisical: grant the consumer's machine identity **read** on the
   `/technitium/` path (one-time).
4. The `lan` zone itself is singleton and owned by
   `homelab-services/services/dns/terraform-config/` — don't create zones
   in consumer repos.

Full copy-pasteable example:
`homelab-platform/docs/examples/new-vm-with-dns/`.

---

## Ansible 2.16+ — `ANSIBLE_SSH_AGENT=auto` Required

When passing `ansible_ssh_private_key_file` (typical for GitHub-Actions-driven
Ansible runs that load the key from Infisical), Ansible 2.16+ tries to add
the key to an SSH agent. If no agent is running, the task fails immediately:

```
[ERROR]: Task failed: Failed to authenticate: Failed to add configured
private key into ssh-agent: Cannot utilize private_key with SSH_AGENT disabled
```

**Fix:** Set `ANSIBLE_SSH_AGENT: "auto"` in the step's `env:` block so Ansible
spawns an agent on demand. Required for every remote-Ansible step.

```yaml
- name: Run playbook against the new VM
  env:
    ANSIBLE_HOST_KEY_CHECKING: "False"
    ANSIBLE_SSH_AGENT: "auto"
  run: ansible-playbook -i "${{ needs.provision.outputs.vm_ip }}," ...
```

---

## Don't Install a Runner on Every VM (or Every Physical Box)

Default pattern for ANY host whose workflows don't need anything host-local:
**drive it via Ansible-over-SSH from terraform-runner**, do NOT install a
GitHub Actions runner on it. Same logic applies whether the host is a
Proxmox VM or a physical machine.

Per-host runners cost you:
- A `GH_PAT` (the default `GITHUB_TOKEN` can't mint registration tokens)
- A separate `register-runner` job that installs the runner service
- PATH issues (`ansible-playbook: command not found` — `uv tool install`
  only adds the bin dir to PATH via `~/.bashrc`, which non-interactive
  runner shells don't source)
- A reinstall on every reprovision
- A chicken-and-egg in `provision-X.yml` when the workflow that provisions
  the host is itself `runs-on: self-hosted-<that-host>` — can't bootstrap
  a host whose runner doesn't exist yet

Use a per-host runner only when **everything** host-local matters: heavy
GPU access required by every step, large local-disk operations that
can't tolerate SSH bandwidth, etc. Even GPU + GGUF management is fine
over SSH (proven by the inference-host refactor — see
`/homelab-local-inference` "Topology" section).

**Decommissioning an orphan runner** registered on a VM/box: delete from
GitHub side via API (`gh api -X DELETE /repos/.../actions/runners/{id}`).
The local systemd unit + actions-runner directory linger and are harmless
clutter. Clean them up by SSH-ing in and `rm -rf ~/actions-runner` if the
runner ran as a regular user; sudo + systemd disable/remove for the unit
file. If the host is going to be reprovisioned soon, leave the local files
— Terraform destroy+rebuild wipes them (VMs only).

Reference workflows in `local-inference-infrastructure/.github/workflows/`:
- `provision-litellm-gateway.yml` — gateway VM (Terraform + Ansible-SSH)
- `provision-inference.yml` — physical inference box (Ansible-SSH)
- `sync-models.yml`, `deploy-alloy.yml` — recurring per-host ops over SSH

All five share the same shape: `runs-on: [self-hosted, self-hosted-infra]`,
load `/ansible` from Infisical, write deploy key, run `ansible-playbook -i
"$IP," ...`, clean up.

---

## DNS Resolution From terraform-runner — `dig @<dns-vm>` at Workflow Time

`terraform-runner` sits on `192.168.1.0/24`; the authoritative `.lan`
resolver (dns-vm) is on `192.168.0.250`. terraform-runner does NOT use
dns-vm as its system resolver, so `.lan` hostnames don't resolve through
the normal stack:

```
ssh: Could not resolve hostname ubuntu-tower.lan: Name or service not known
```

This bit the inference-host refactor on first try. Two options:

**Don't**: reconfigure terraform-runner's `/etc/systemd/resolved.conf` to
use dns-vm. Works but you're now silently dependent on dns-vm for every
DNS lookup the runner makes (apt, docker pulls, GitHub API, etc.).

**Do**: have the workflow resolve `.lan` names against dns-vm explicitly,
just before the ansible step:

```yaml
- name: Resolve <hostname> to IP via dns-vm
  id: target
  run: |
    IP=$(dig @192.168.0.250 +short "$HOSTNAME" A 2>/dev/null | head -n1)
    if [ -z "$IP" ]; then
      IP=$(nslookup "$HOSTNAME" 192.168.0.250 2>/dev/null | awk '/^Address: 192\./ {print $2; exit}')
    fi
    [ -n "$IP" ] || { echo "ERROR: $HOSTNAME did not resolve"; exit 1; }
    echo "ip=$IP" >> "$GITHUB_OUTPUT"

- name: Run ansible
  run: |
    ansible-playbook -i "${{ steps.target.outputs.ip }}," ...
```

DHCP renumbering becomes transparent — every run gets a fresh lookup.
`dig` is preferred; `nslookup` fallback covers the case where
`bind9-dnsutils` isn't installed on the runner.

The manifest / inventory keeps `.lan` hostnames as the source of truth.
Only the SSH target changes (resolved IP instead of FQDN).

---

## Ad-Hoc Ansible Inventory `-i "<host>,"` Requires `hosts: all`

Common pattern in CI: pass the target via `-i "<ip-or-host>,"` (the
trailing comma is what tells ansible "this is an inline inventory, not
a file"). The single host lands in the `all` and `ungrouped` groups —
NOT in any named group.

If the playbook header is:
```yaml
- name: My playbook
  hosts: inference_machines
```

…it'll silently no-op the entire run with:
```
[WARNING]: Could not match supplied host pattern, ignoring: inference_machines
PLAY RECAP *********
(no PLAY RECAP rows, 0 hosts matched)
```

**And the job will report "success"** because zero-hosts-matched isn't
a failure to ansible. Real bug, silent symptom.

**Fix:** use `hosts: all` in playbooks that are invoked via inline
inventory:

```yaml
- name: My playbook
  hosts: all   # works with -i "host," AND with inventory.ini group filters
```

Local-debug runs with the inventory file still work because `all`
implicitly includes every host in the inventory.

---

## `gather_facts: true` When You Use `ansible_env.*`

When a playbook references `{{ ansible_env.HOME }}` or any other fact
(`ansible_user_id`, `ansible_distribution`, etc.), facts must be gathered
first. If you disabled fact-gathering for speed:

```yaml
- hosts: all
  gather_facts: false   # ← culprit
  tasks:
    - stat:
        path: "{{ ansible_env.HOME }}/.lmstudio/bin/lms"   # FAILS
```

…every task that touches `ansible_env` fails with:
```
Finalization of task args failed: Error while resolving value for 'path':
'ansible_env' is undefined
```

**Fix:** either set `gather_facts: true` (the default — just delete the
override), or pass the value via extra-vars instead of a fact:

```yaml
gather_facts: true
# …or:
# -e remote_home=/home/blake
```

---

## Make Provisioning Playbooks Self-Skip on Already-Done Phases

For playbooks that should be safe to re-run on already-provisioned hosts
AND to provision fresh ones, gate each disruptive phase on a cheap
precheck rather than relying on each task's own idempotency. Two patterns
from `inference-setup.yml`:

**Skip disk-expand on physical boxes without cloud-image LVM:**

```yaml
- name: "[disk] Precheck: does /dev/ubuntu-vg/ubuntu-lv exist?"
  stat:
    path: /dev/ubuntu-vg/ubuntu-lv
  register: lv_stat
  tags: disk

- name: "[disk] Extend LV"
  command: lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
  when: lv_stat.stat.exists
  tags: disk
# …same when: on resize2fs
```

A bare-metal install without LVM gets a "skipped" message instead of an
`lvextend` failure.

**Skip NVIDIA driver install + reboot when driver already works:**

```yaml
- name: "[drivers] Precheck: nvidia-smi works?"
  command: nvidia-smi
  register: nvidia_smi_precheck
  ignore_errors: true
  changed_when: false
  tags: drivers

- name: "[drivers] Install ubuntu-drivers-common"
  apt: { name: ubuntu-drivers-common, state: present, update_cache: true }
  become: true
  when: nvidia_smi_precheck.rc != 0
  tags: drivers
# …same when: on each driver-install and reboot task
```

A box that already has a working driver gets a fast no-op; a fresh box
installs, reboots, and the playbook intentionally fails with a "REBOOT
IN PROGRESS — re-run with skip_tags=disk,drivers" message so the workflow
operator knows to dispatch a resume.

The cost is ~3 added tasks per phase. The benefit is one playbook for
fresh-vs-existing and confidence to re-run after every commit.

---

## Best-Effort Boot-Time Side Effects

Systemd `ExecStartPost` and ansible tasks that do "warm-start"
optimizations (preloading a model, warming a cache) should be tolerant
of missing prerequisites — otherwise they block fresh-box provisioning
on something that's downloaded later by a different workflow.

**Systemd:** prefix the line with `-` to ignore exit codes:

```ini
[Service]
ExecStartPost=/path/lms server start ...
-ExecStartPost=/path/lms load qwen3.5-9b -c 32768 -y   ← `-` makes it best-effort
```

**Ansible:** `failed_when: false`, register the result, optionally emit a
debug note:

```yaml
- name: "[lmstudio] Preload reasoning model (best-effort)"
  shell: "{{ ansible_env.HOME }}/.lmstudio/bin/lms load qwen3.5-9b -c 32768 -y"
  register: lms_load_result
  changed_when: lms_load_result.rc == 0
  failed_when: false

- name: "[lmstudio] Note if preload was skipped"
  debug:
    msg: "{{ 'Preload OK' if lms_load_result.rc == 0 else 'Skipped (model not downloaded yet — sync-models will fetch).' }}"
```

The truthful version of "skipped" beats a red-X job for a step the
operator never intended to be a gate.

---

## One-Time Deploy-Key Bootstrap for New Ansible-SSH Hosts

When adding a new host to the Ansible-over-SSH cluster (gateway, inference,
whatever), the Ansible deploy public key needs to land in the target user's
`~/.ssh/authorized_keys` BEFORE any workflow can SSH in. Pattern from
`local-inference-infrastructure/scripts/bootstrap-deploy-key.sh`:

1. Pull `ANSIBLE_PRIVATE_KEY` from Infisical (`/ansible`, prod) via
   `infisical secrets get ANSIBLE_PRIVATE_KEY --plain` into `mktemp(1)`.
2. Derive the public key locally: `ssh-keygen -y -f /tmp/key > /tmp/key.pub`.
3. `ssh-copy-id -i /tmp/key.pub user@new-host.lan` (prompts for password
   once; idempotent — a re-run skips if the key already works).
4. `trap rm -f /tmp/key /tmp/key.pub EXIT` so nothing leaks even on Ctrl-C.

Reusable, idempotent, no secrets persisted to disk outside `/tmp`. Works
for any host whose `<user>` has password auth enabled at least once.

Don't try to drive this from a workflow — the workflow ITSELF needs the
key to be in place. It's a strictly local one-time op (a human runs the
script from a shell where they're logged into Infisical).

---

## `git add -A` in User-Shared Working Trees Sweeps WIP

When committing inside a repo the user actively works in (i.e. almost any
homelab repo), use `git add -- <specific-paths>` rather than `git add -A`.
The user often has uncommitted WIP in the working tree (in-progress
playbooks, vars, etc.); `git add -A` pulls those into a commit whose
message doesn't describe them, and the no-PR direct-push policy on these
repos means there's no review gate to catch it.

**Habit:**
```bash
git status --short                    # confirm what's modified
git add -- path/I/edited.yml ...      # explicit
git status --short                    # confirm staged set matches intent
git commit -m "..."
```

`git add -A` is fine in fresh worktrees or repos with no human
collaborator in parallel.

---

## Per-Repo Runner Registration

`terraform-runner` (label `self-hosted-infra`) needs to be **registered
separately for every repo** it serves. Registration is per-repo, not
per-org for personal accounts.

When you add a new repo whose workflows target `[self-hosted, self-hosted-infra]`,
dispatch `homelab-platform/.github/workflows/register-runner.yml` against
the new repo URL — otherwise jobs queue forever. The runner can be registered
to N repos simultaneously; each gets its own `actions-runner-<repo>`
directory and its own systemd unit.

---

## Infisical-action — Pin to v1.0.15 (last Node 20)

`Infisical/secrets-action@v1.0.16` was retagged April 2026 to switch from
Node 20 to Node 24. Older self-hosted runner binaries (2.322.0 and below)
don't support Node 24:

```
##[error]Can't use 'using: node24' as it's not supported in the runtime
```

**Fix:** Pin to `@v1.0.15` in every workflow on every repo until the
self-hosted runner binary is upgraded.

```yaml
- uses: Infisical/secrets-action@v1.0.15
```

**Long-term fix:** Upgrade actions-runner to a version that ships Node 24
(2.324+), then unpin.

---

## Multi-Project Infisical Pattern

Default homelab setup uses **two Infisical projects** per repo that needs
secrets:

- **Shared infra project** (`homelab-platform-eu2-m`): holds `/proxmox`,
  `/terraform`, `/ansible` — used by every repo that calls the reusable
  `provision-vm.yml` workflow. Referenced via `vars.INFISICAL_PROJECT_SLUG`.
- **Per-repo project** (e.g. `local-inference-infrastructure-pca-l`): holds
  app-scoped secrets like `/litellm`. Referenced via a repo-specific variable
  like `vars.LITELLM_INFISICAL_PROJECT_SLUG`.

The **same OIDC machine identity** (`vars.INFISICAL_IDENTITY_ID`) is bound
to both projects, so only the `project-slug` differs between secret-load
steps. Trust binding on the identity must include both projects — granted
in the Infisical UI under the identity's "Project Access" tab.

```yaml
# Shared infra path
- uses: Infisical/secrets-action@v1.0.15
  with:
    identity-id:  ${{ vars.INFISICAL_IDENTITY_ID }}
    project-slug: ${{ vars.INFISICAL_PROJECT_SLUG }}
    secret-path:  /ansible

# Per-repo path
- uses: Infisical/secrets-action@v1.0.15
  with:
    identity-id:  ${{ vars.INFISICAL_IDENTITY_ID }}
    project-slug: ${{ vars.LITELLM_INFISICAL_PROJECT_SLUG }}
    secret-path:  /litellm
```

---

## Verifying Loaded Secrets Without Leaking Them

When debugging "extra-var came through empty," **don't** echo the value.
Echo the length:

```bash
for v in LITELLM_MASTER_KEY LITELLM_POSTGRES_PASSWORD; do
  len=$(eval echo \${#$v})
  [ "$len" -eq 0 ] && echo "MISSING: $v" || echo "OK: $v ($len chars)"
done
```

Zero-length means the secret-path doesn't contain that key, or the key
exists in a different env / project. Non-zero confirms the action loaded
it without exposing the value in logs. Pair with the `length-only` debug
pattern in the Infisical/Terraform section above.

---

## `gh run rerun --failed` Uses the Original SHA's Workflow File

Counterintuitive: re-running a failed job does **not** pick up workflow
edits made after the original run started. The workflow YAML is pinned to
the head SHA of the original dispatch.

When testing a workflow fix, always do a **fresh** `gh workflow run` after
the commit; don't `--failed`-rerun.

---

## `infra-runner.yml` — `svc.sh` Guard When Re-Registering

The `runner` tag in `homelab-platform/ansible/infra-runner.yml` historically
ran `./svc.sh stop` and `./svc.sh uninstall` unconditionally. On a fresh
runner directory those scripts don't exist yet, and the task fails:

```
/bin/sh: 1: ./svc.sh: not found  (exit 127)
```

`failed_when` didn't catch "not found" — only "not installed". The fix
(now upstream in `homelab-platform`) is a `stat` check before stop/uninstall:

```yaml
- name: "[runner] Check whether svc.sh exists"
  stat:
    path: "{{ github_actions_runner_dir }}/svc.sh"
  register: runner_svc_script
  tags: runner

- name: "[runner] Stop existing service if present"
  shell: ./svc.sh stop
  args: { chdir: "{{ github_actions_runner_dir }}" }
  become: true
  when: runner_svc_script.stat.exists
  ...
```

Any repo that forks or copies `infra-runner.yml` needs the same guard.

---

## Bind-Mounted Config Files That Don't Exist Yet

Pattern in `litellm-gateway-setup.yml`: the LiteLLM container is started
with `command: --config /app/litellm/litellm-config.yaml`, where the config
file is bind-mounted from `~/litellm/`. But the config is generated and
shipped by a **separate** workflow (`deploy-litellm.yml`) on every push.

On a brand-new VM, the provision playbook leaves the container in a
crash-loop until `deploy-litellm` runs at least once. This is expected
behavior — the first `deploy-litellm` after provisioning fixes it.

When adding a similar pattern: document the two-step bring-up explicitly,
or have the provision workflow trigger the deploy workflow at the end
via `gh workflow run`.

---

## `gh --jq` Instead of External `jq` in Monitor/Background Bash

When monitoring GitHub Actions runs from Claude Code on Windows via the
`Monitor` tool, the background bash environment does **not** have `jq` on
PATH. Loops that pipe through `jq` silently produce empty output, and
events never fire.

**Fix:** Use `gh`'s built-in `--jq` flag instead of an external `jq`:

```bash
gh run view $ID --json status,conclusion,jobs \
  --jq '.jobs[] | "\(.name): \(.status)/\(.conclusion)"'
```

Avoid `... | jq` in any script you intend to run inside the `Monitor` tool
or as a background Bash on a Windows host.

---

## dns-record Module — Prefer CNAME for VM Aliases

When using the `dns-record` module from homelab-platform to create a
friendly alias for a VM (e.g. `dashboard.lan` for `dashboard-vm`), use
the **CNAME mode** pointing at the VM's auto-registered hostname, not
the A-record mode pinned at `module.vm.vm_ip`:

```hcl
module "dns" {
  source   = "github.com/BlakeHastings/homelab-platform//terraform/modules/dns-record?ref=main"
  hostname = "dashboard"                       # → dashboard.lan
  cname    = "${module.vm.vm_name}.lan"        # → dashboard-vm.lan
  zone     = "lan"
}
```

**Why:** The router auto-registers each VM's hostname into Technitium as
a TTL-900 A record on DHCP lease (visible as `<vm_name>.lan` in
`records lan`). When the VM's IP changes (DHCP renewal without a
reservation), the router pushes the new A record automatically. A CNAME
pointing at `<vm_name>.lan` follows for free; a Terraform-managed A
record pinned at `module.vm.vm_ip` does not, and goes stale until
`terraform apply` runs again.

**Symptom of the wrong choice:** `dashboard.lan` returns the wrong IP
after a DHCP renewal while `dashboard-vm.lan` resolves correctly. Fix is
to switch from `ip =` to `cname =` and re-provision.

**When the A-record mode IS correct:**
- DHCP reservation on the router pins the MAC → IP (record stays valid)
- Target is not a VM (Proxmox host, external service) — no router
  auto-registration to chain off (use `terraform-config/a_records` map)
- Static IP assignment

Full module docs: `homelab-platform/terraform/modules/dns-record/README.md`.

---

## GitHub Actions — Zombie "In-Progress" Runs

Distinct from the runner-`offline` failure mode above. Symptom:

- Runner status shows `online`, `busy: true`
- Specific run shows `in_progress` for many minutes
- Step-level `gh run view <id> --json jobs` shows every step has
  `status: completed, conclusion: success` — yet the run-level state
  remains `in_progress`
- New queued runs cannot start (runner is "busy" with the zombie)

This is a stale-state bug where the runner's "I finished" heartbeat
never reached GitHub (network blip, runner restart, OOM kill of the
runner process mid-cleanup, etc.). The actual work is done; only the
control-plane status is wrong.

**Recovery:**

```bash
# 1. Confirm steps actually completed
gh run view <ID> --json jobs --jq '.jobs[0].steps[] | {name, status, conclusion}'

# 2. Cancel the zombie (this often won't propagate to the runner,
#    but flips the API state so queued runs can proceed)
gh run cancel <ID>

# 3. SSH into the runner VM and restart the systemd unit (same as
#    the offline-recovery pattern above)
sudo systemctl restart actions.runner.OWNER-REPO.RUNNER.service
```

After step 3, the zombie flips to `failure`/`cancelled` and queued
runs start within seconds.

**Diagnostic shortcut:** if all steps show `completed/success` but the
run-level conclusion is empty/null and time-since-last-step exceeds
~2 minutes, it's a zombie. Don't wait it out — restart the runner.

---

## Terraform — Output Description Fields Don't Allow `${}` Interpolation

Subtle but recurring. Bites at plan time with a confusing message:

```
Error: Variables not allowed
  on main.tf line 35, in output "vm_ip":
   35:   description = "VM IP, also reachable as ${module.dns.fqdn}"
Variables may not be used here.
```

Output `description` (and `variable` `description`) fields must be
**literal strings** — Terraform evaluates them statically for `terraform
output -json` introspection, before any module wiring is resolved.

**Fix:** make the description a plain string. Reference the related
value as a separate output so consumers can introspect both:

```hcl
output "vm_ip" {
  description = "VM IP — also reachable by FQDN from the fqdn output below."
  value       = module.vm.vm_ip
}

output "fqdn" {
  description = "Friendly alias FQDN (a CNAME to the router-registered hostname)."
  value       = module.dns.fqdn
}
```

Same rule applies to:
- `variable "x" { description = "..." }` — no interpolation
- `output "x" { description = "..." }` — no interpolation
- Most resource attribute `description` fields (varies by provider)

Interpolation IS allowed in `value`, `default`, and most other fields.

---

## `localhost` Inside a Bridge-Networking Container is the Container, Not the Host

Surprising every time. A container in default Docker bridge networking
that does `curl http://localhost:3100` hits **the container itself**, not
the host's 3100. So:

- Prometheus container scraping `localhost:9100` — scrapes Prometheus, not
  the host's Node Exporter.
- Alloy container pushing to `http://localhost:3100/loki/api/v1/push` —
  pushes to Alloy itself, not the host's Loki.

The host's bound port (`-p 3100:3100`) is reachable from inside the
container at the host's LAN IP / hostname, **not** at `localhost`. Two
right ways:

1. **Hostname** — `<host>.lan:<port>` (resolves to host's LAN IP, reaches
   the published Docker port). Preferred — survives DHCP and the same
   config works from other hosts too.
2. **`host.docker.internal`** — Docker's special hostname for the host
   gateway. On Linux requires `extra_hosts: ["host.docker.internal:host-gateway"]`
   in compose. Less portable.
3. **Host networking** — `network_mode: host` puts the container ON the
   host's network namespace. Then `localhost` IS the host. Node Exporter
   uses this pattern (it must, to read host metrics). For most services
   it's overkill and removes the published-port isolation.

If a stack works fine when you `curl` from outside the VM but fails when
one container talks to another, suspect this. Especially common for
Prometheus self-scrapes and Alloy pushing to a local Loki.

---

## `docker compose up -d` Won't Recreate a Container After a Bind-Mount File Change

Compose's `up -d` is idempotent on its *own model* — image, env, ports,
volumes-as-paths. It does **not** look at the content of files inside
bind-mounted volumes. So if you:

1. Change a config file on the host that's bind-mounted into a container
2. Run `docker compose up -d`

…the container stays unchanged, still reading the old in-memory config.
Same gotcha if the container CrashLooped from a prior bad config — `up
-d` sees it as "already there, restart-policy will keep handling it" and
leaves it.

**Force recreation:**

```bash
docker compose up -d --force-recreate
```

…recreates every container in the compose file even when nothing in the
compose model changed. The cost is 5-30s of downtime per service. For
deploys triggered by config-file edits (e.g. our `deploy-observability`),
this is the right default — guarantees the new config is actually in
effect.

Alternative for surgical recreation: `docker compose up -d --force-recreate <service>`.

---

## Ansible `community.docker.docker_container` Idempotency Doesn't See Bind-Mount Files

Parallel of the docker-compose gotcha. `docker_container` checks the
container's image, env, ports, volume paths — but **not** the contents of
the files at those volume paths. So if a `template:` task updates
`~/alloy/config.alloy` (bind-mounted into the container), the subsequent
`docker_container` task with `state: started` sees no change and does
nothing.

**Fix: handler triggered from the template task.**

```yaml
- name: Render config
  template:
    src: foo.j2
    dest: ~/foo/config.yaml
  notify: restart foo

- name: Run container
  community.docker.docker_container:
    name: foo
    image: foo:latest
    volumes:
      - ~/foo/config.yaml:/etc/foo/config.yaml
    # ... (idempotent on its own parameters only)

handlers:
  - name: restart foo
    community.docker.docker_container:
      name: foo
      state: started
      restart: true   # stop + start; container re-reads the bind-mount on boot
```

Caveat: handlers only fire when the notifying task reports `changed`.
If the on-disk file already matches what the template would render (e.g.
manual edit, or previous run already applied the new value), no diff →
no notify → no restart. For pathological cases (state drift between disk
and in-memory), add a manual `docker restart <name>` step or run the
play with `--force-handlers`.

---

## `docker_container` Is Additive: It Never Reaps Containers You Stop Declaring

`community.docker.docker_container` only manages the containers a playbook
*currently declares*. Delete a container's task (or move the service to a
different host) and re-running the playbook does **nothing** to the old
container — it keeps running, orphaned, because nothing references it
anymore. There is no `state: present` reconcile-the-whole-host semantic
the way `docker compose down` removes services dropped from the compose
file. Re-applying config is not a cleanup mechanism.

**Consequence for decommissioning:** "I removed it from the playbook, so
re-running provisioning will clean the box" is wrong. To actually remove a
service you have two choices:

- An explicit `state: absent` task (the durable, reproducible option;
  keep it in the playbook so any future re-provision is self-healing).
  See the legacy-Playwright cleanup block in
  `local-inference-infrastructure/ansible/litellm-gateway-setup.yml` for
  the pattern.
- A one-off `docker rm -f <name>` over SSH (fast, but not captured in
  config, so the orphan can reappear if anything ever re-runs the old path).

Watch for **dependency stragglers**: removing `open-webui` and `litellm`
leaves `litellm-postgres` (their backing DB) running untouched, because it
was its own `docker_container` task. Always `docker ps -a` after a manual
removal and reconcile against what the playbook *should* leave behind.

---

## systemd-resolved Drop-In for Routing One Domain to One DNS Server

Ubuntu 24.04 uses systemd-resolved by default. To route only `.lan`
queries to Technitium (192.168.0.250) while keeping the system's default
resolvers for everything else, write a drop-in:

```ini
# /etc/systemd/resolved.conf.d/technitium.conf
[Resolve]
DNS=192.168.0.250
Domains=~lan
```

Then `sudo systemctl restart systemd-resolved`. The `~lan` syntax marks
the domain as **routing-only** — systemd-resolved sends only `*.lan`
queries to that DNS server, and other queries continue to the system's
configured resolvers.

This pattern is the right answer for "give this device `.lan` resolution
without changing its primary DNS" — used today in
`homelab-services/.github/workflows/deploy-observability.yml` (for the
runner) and centralized in `homelab-platform/ansible/base-vm.yml` (for
every provisioned VM).

The dropin file persists across reboots and survives cloud-init
re-runs (cloud-init writes `/etc/netplan/...`, not resolved.conf.d).

---

## Container User UIDs for Bind-Mount Permissions

When a containerized service writes to a bind-mounted host directory, the
host directory must be owned by the UID the container runs as. The
process inside the container can't see host usernames — only numeric
UIDs map across the namespace boundary. If perms are wrong the container
CrashLoops on startup ("permission denied" writing to its data dir).

**Reference table for the services we run:**

| Image | UID | GID |
|-------|-----|-----|
| `grafana/grafana` | 472 | 472 |
| `prom/prometheus` | 65534 (nobody) | 65534 |
| `grafana/loki` | 10001 | 10001 |
| `grafana/tempo` | 10001 | 10001 |
| `technitium/dns-server` | 0 (root) | 0 |
| `nginx/nginx` | 0 (root, drops to 101 nginx) | 0 |
| `gethomepage/homepage` | 0 (root) | 0 |

**Idiomatic deploy fix** — in the workflow's `mkdir` step, also `chown`:

```bash
ssh ubuntu@$VM_IP <<EOSH
  mkdir -p ~/services/observability/data/{grafana,prometheus,loki,tempo}
  sudo chown -R 472:472     ~/services/observability/data/grafana
  sudo chown -R 65534:65534 ~/services/observability/data/prometheus
  sudo chown -R 10001:10001 ~/services/observability/data/loki
  sudo chown -R 10001:10001 ~/services/observability/data/tempo
EOSH
```

Idempotent — re-chown is a no-op once ownership is correct.

To find a new image's UID: `docker run --rm --entrypoint id <image>` —
output line is `uid=NNN(name) gid=NNN(name) ...`.

---

## GitHub Actions: `workflow_dispatch` 422 Means the YAML Won't Parse

Symptom:

```
could not create workflow dispatch event: HTTP 422:
Workflow does not have 'workflow_dispatch' trigger
(https://api.github.com/repos/OWNER/REPO/actions/workflows/<id>/dispatches)
```

Misleading. The workflow DOES have `on: w

…(truncated)