Homelab Observability

Homelab Observability

The LGTM stack on observability-vm. Grafana UI: http://observability-vm.lan:3000 (admin/changeme on first login). Source of truth: homelab-services/services/observability/.

DNS: observability.lan is a Terraform-managed CNAME → observability-vm.lan (router-auto-registered, follows DHCP). All clients ship to observability-vm.lan:3100 and survive DHCP renewals — see [[feedback-dns-record-cname-pattern]].

What runs where

On every VM (installed by homelab-platform/ansible/base-vm.yml)
  ├── Node Exporter (host network)    :9100   → metrics, scraped by Prometheus
  └── Grafana Alloy (bridge)          :12345  → ships container logs + journal to Loki
                                              (bound to 127.0.0.1 only — debug UI)

On observability-vm (docker-compose in services/observability/)
  ├── Grafana        :3000           → unified UI, dashboards, alerting
  ├── Prometheus     :9090           → metrics store (bound 127.0.0.1)
  ├── Loki           :3100           → log store (bound 0.0.0.0)
  └── Tempo          :3200, 4317-4318 → traces (HTTP API + OTLP gRPC + OTLP HTTP)

Data sources + the Node Exporter Full dashboard auto-load on Grafana startup — see "Declarative provisioning" below.

File layout

services/observability/
├── docker-compose.yml            # LGTM stack
├── prometheus.yml                # scrape targets — add VMs here
├── grafana/
│   └── provisioning/
│       ├── datasources/
│       │   └── datasources.yaml  # Prometheus, Loki, Tempo (pinned UIDs)
│       └── dashboards/
│           ├── dashboards.yaml   # file-provider — scans for .json
│           └── node-exporter-full.json  # 31-panel dashboard (id 1860)
└── terraform/                    # VM provisioning + DNS CNAME

Adding a Prometheus scrape target

Edit services/observability/prometheus.yml, add an entry under scrape_configs. Use the .lan hostname (resolves via Technitium, survives DHCP):

- job_name: 'my-new-vm'
  static_configs:
    - targets: ['my-new-vm.lan:9100']
      labels:
        node: 'my-new-vm'

Push → Deploy observability stack auto-triggers → Prometheus container restarts → new target appears within ~15s. Verify:

curl -s -u admin:changeme \
  "http://observability-vm.lan:3000/api/datasources/proxy/uid/prometheus/api/v1/targets?state=any" \
  | python -c "import sys,json; [print(f'  {t[\"labels\"][\"job\"]:20} {t[\"health\"]}') for t in json.load(sys.stdin)['data']['activeTargets']]"

For GPU nodes add a second entry on :9400 for DCGM exporter (not installed by base-vm.yml — needs project-specific Ansible).

When the service is behind UFW

Node Exporter (:9100) and DCGM (:9400) on every VM accept any LAN source, so adding a scrape job is enough. But application-tier /metrics endpoints often sit behind a tight UFW rule that only allows the service's primary client (e.g. llama-server on the inference hosts is locked to the gateway IP 192.168.0.5). Adding a scrape job while UFW still blocks observability-vm produces a target that shows health=down with connection refused / i/o timeout in Grafana, despite the service responding fine to its real client.

The fix: in the playbook that opens the service's port to its primary client, add a parallel ufw allow rule with from_ip: {{ observability_server_ip }} (a var pinned to 192.168.0.23 in each project's ansible/vars/main.yml). Example from local-inference-infrastructure/ansible/inference-setup.yml:

- name: "[llama-server] UFW: allow gateway → llama-server port"
  community.general.ufw:
    rule: allow
    from_ip: "{{ litellm_gateway_ip }}"      # primary caller
    port: "{{ item.engine_args.port | default(8080) }}"

- name: "[llama-server] UFW: allow observability-vm → llama-server port"
  community.general.ufw:
    rule: allow
    from_ip: "{{ observability_server_ip }}" # Prometheus scrape
    port: "{{ item.engine_args.port | default(8080) }}"

Same pattern for TTS gateway, SearXNG, Playwright, future services. The moment you add a scrape job for a port that's UFW-restricted, audit whether the playbook also lets observability-vm in.

Don't be misled by local repo configs/prometheus/ folders

local-inference-infrastructure/configs/prometheus/scrape-targets.yaml isn't loaded by anything — it's a stale seed file from the pre-May-2026 single-machine era. The first lines literally read "Add this block to your central Prometheus prometheus.yml under scrape_configs:". The only file that actually controls scrape config is homelab-services/services/observability/prometheus.yml. If you find a similar configs/prometheus/ folder in another homelab repo, verify it's not also dead documentation before editing.

Adding a Grafana dashboard (declarative)

1. Find or build a dashboard. For grafana.com dashboards:

   curl -sL -o services/observability/grafana/provisioning/dashboards/<name>.json \
     "https://grafana.com/api/dashboards/<ID>/revisions/latest/download"
   

2. Replace data source placeholders. Provisioned dashboards don't go through the UI's "Import" templating step, so ${DS_PROMETHEUS}-style placeholders stay literal and break the panels. Pin to the data source UIDs we provision (prometheus, loki, tempo):

   python -c "
   import json, pathlib
   p = pathlib.Path('services/observability/grafana/provisioning/dashboards/<name>.json')
   j = json.loads(p.read_text())
   s = json.dumps(j).replace('\${DS_PROMETHEUS}', 'prometheus').replace('\${DS_LOKI}', 'loki').replace('\${DS_TEMPO}', 'tempo')
   j = json.loads(s)
   j['id'] = None   # required: Grafana assigns a new id on load
   p.write_text(json.dumps(j, indent=2))
   "
   

3. Commit + push. Deploy observability stack auto-triggers; Grafana's file provider sees the new JSON within 30s and loads it.

The Node Exporter Full dashboard (node-exporter-full.json) is the reference example — Prometheus UID prometheus, id set to null, panels rendering without UI interaction.

Adding a data source

Edit grafana/provisioning/datasources/datasources.yaml. Pin the UID so future dashboards can hardcode it.

- name: My DB
  uid: mydb
  type: postgres
  access: proxy
  url: my-db.lan:5432
  jsonData: {sslmode: disable}
  secureJsonData: {password: ${MY_DB_PASSWORD}}   # use HOMEPAGE-style env injection

Restart Grafana (docker compose restart grafana) or just dispatch deploy-observability — the --force-recreate step picks it up. Data source visible in Connections → Data Sources immediately, no UI clicks.

For secrets — secureJsonData accepts Grafana's env-var interpolation ${ENV_VAR} at runtime. The plumbing for /grafana/ secrets in Infisical isn't wired up yet (Phase 4 from the obs setup).

Container UIDs (data dir ownership)

LGTM containers run as non-root and bind-mount data dirs. Wrong perms → CrashLoop on startup. Deploy workflow chowns each one:

If you add a service that crashloops on first deploy, this is the first thing to check. See deploy-observability.yml for the chown pattern.

Alloy — log shipping config

Template: homelab-platform/ansible/templates/alloy-config.j2. Rendered per-VM by Ansible at provision time. Key bits:

• node label uses {{ vm_name | default(ansible_hostname) }} so Loki streams are labeled by the VM's authoritative name (passed from the workflow input), not the Alloy container's random hostname AND not the VM's drifted system hostname. Three earlier bugs hit this surface:
  1. Original template used constants.hostname → in bridge networking, that's the random container ID → all VMs labeled with hex strings.
  2. First fix used ansible_hostname → but searxng-gateway's /etc/hostname had drifted to terraform-runner → its logs landed under a wrong VM's label silently.
  3. Current fix uses vm_name from the workflow input → always correct. See homelab-bootstrap skill's "ansible_hostname can drift" section for the three-layered defense (workflow → hostname enforcement → template fallback).
• Loki endpoint comes from the observability_server_ip workflow input — pass observability-vm.lan (hostname-based, survives DHCP) rather than an IP literal.
• Restart on config change: base-vm.yml's template task notifies the restart alloy handler. The container doesn't auto-restart on bind-mount file content changes (see homelab-bootstrap skill's "Ansible docker_container idempotency" section).

Inference hosts use a separate Alloy config at local-inference-infrastructure/configs/alloy/alloy-config.alloy.j2 — physical machines (ubuntu-tower, ubuntu-tower-02), not Proxmox VMs, so they're outside the base-vm.yml flow. That config is rendered by local-inference-infrastructure/ansible/deploy-alloy.yml (uses template: — must do, since the file has Jinja placeholders) and dispatched by the matrix-based .github/workflows/deploy-alloy.yml. The node label there is set to the matrix host's logical name (inference-01, inference-02) — same drift-resistant pattern as the VM vm_name approach. Gotcha that bit us: the original file was named .alloy and copied with copy:, shipping literal placeholders verbatim. See homelab-bootstrap skill's "copy: ships files verbatim" section.

To verify Alloy is shipping correctly-labeled from a given VM:

curl -s -u admin:changeme \
  "http://observability-vm.lan:3000/api/datasources/proxy/uid/loki/loki/api/v1/label/node/values" \
  | python -c "import sys,json; print(json.load(sys.stdin).get('data'))"

Expected: a list of distinct VM names matching what's provisioned. Red flags:

• Random hex strings = container IDs leaking through (template using constants.hostname)
• Wrong VM name = hostname drift (template using ansible_hostname while OS hostname is wrong)
• A VM in the expected list is missing volume = it might be shipping under one of the other labels you DO see; cross-check with count_over_time({node="<suspect>"}[5m]) (see "Debugging missing-looking telemetry" below)

DNS for .lan from inside VMs

Alloy on each VM needs to resolve observability-vm.lan. base-vm.yml writes a systemd-resolved drop-in (/etc/systemd/resolved.conf.d/technitium.conf) that routes only .lan queries to Technitium:

[Resolve]
DNS=192.168.0.250    # dns-vm; on dns-vm itself this is 127.0.0.1
Domains=~lan         # only .lan queries go here; everything else uses defaults

If a VM was provisioned before this dropin was added, re-provision to apply. The dropin is idempotent — Ansible's template task triggers a restart systemd-resolved + restart alloy handler only when content changes.

Tempo config — known gotcha

The compose file's embedded Tempo config used to have a compactor: block. Current Tempo rejects it:

field compactor not found in type app.Config

Removed in services/observability/docker-compose.yml. Tempo uses defaults — fine for now. If you tune compaction later, the field probably moved under a different parent or got renamed; check Tempo upstream docs for current schema.

Same shape of error would apply to other LGTM components — when an upstream image rejects a config field, the fastest path is "remove the block, run with defaults, revisit when you actually need to tune it."

Deploy workflow shape

.github/workflows/deploy-observability.yml does more than the obvious. Reads top-to-bottom:

1. Checkout
2. Ensure runner can resolve .lan — writes the same systemd-resolved dropin on terraform-runner itself; necessary because workflows ssh to <vm>.lan later
3. Load Ansible secrets from Infisical
4. Get VM IP from Terraform state — soon-to-be-legacy; future deploys should use observability-vm.lan directly
5. Write SSH key
6. Create service + data dirs + chown — runs BEFORE rsync (the rsync-before-mkdir bug from homelab-bootstrap skill)
7. Sync service files via rsync
8. Pull and start stack — docker compose pull && docker compose up -d --force-recreate (force-recreate ensures bind-mount edits take effect)
9. Wait + report container state — sleeps 15s, runs ps -a, tails logs from any non-running service. Also probes /api/v1/targets and prints health + lastError for auth'd scrape jobs (pve, litellm). Container-running ≠ scrape-working — a 403 on the upstream API leaves the container "healthy" but the metric pipeline broken. When the pve job is unhealthy, the step also dumps docker compose logs --tail 40 pve-exporter because the exporter's wrapped exception names the actual Proxmox 4xx + path (e.g. Permission check failed (/, Sys.Audit))
10. Refresh Alloy on known client VMs — SSH to dns-vm.lan, dashboard-vm.lan, observability-vm.lan; restart alloy on each; tail post-restart logs
11. Cleanup SSH key

The "client VMs" list in step 10 is static — add new VMs there as they come online with Alloy.

First-time admin password (not yet declarative)

Hardcoded as changeme in docker-compose.yml. Change via UI on first login.

Phase 4 follow-up: move to Infisical at /grafana/ path, inject via env var. Deploy workflow loads it, sets GF_SECURITY_ADMIN_PASSWORD at container start. Then a tear-down + redeploy recreates with the correct password automatically.

Quick verification commands

# Stack health
for svc in "grafana:3000/api/health" "prometheus:9090/-/ready" "loki:3100/ready" "tempo:3200/ready"; do
  name=$(echo "$svc" | cut -d: -f1)
  path=$(echo "$svc" | cut -d: -f2-)
  status=$(curl -s -o /dev/null -w "%{http_code}" -m 5 "http://observability-vm.lan:$path" 2>&1)
  echo "$name → $status"
done

# Scrape targets
curl -s -u admin:changeme \
  "http://observability-vm.lan:3000/api/datasources/proxy/uid/prometheus/api/v1/targets" \
  | python -c "import sys,json; [print(f'  {t[\"labels\"][\"job\"]:20} {t[\"scrapeUrl\"]:50} {t[\"health\"]}') for t in json.load(sys.stdin)['data']['activeTargets']]"

# Log volume per node, last 5min
curl -s -u admin:changeme --get \
  "http://observability-vm.lan:3000/api/datasources/proxy/uid/loki/loki/api/v1/query" \
  --data-urlencode 'query=sum by (node, job) (count_over_time({node=~".+"}[5m]))' \
  | python -c "import sys,json; [print(f'  node={x[\"metric\"][\"node\"]:20} job={x[\"metric\"][\"job\"]:10} count={x[\"value\"][1]}') for x in json.load(sys.stdin)['data']['result']]"

Debugging missing-looking telemetry — check labels first

Before assuming a node isn't shipping, list the actual label values:

# Loki: what node names exist?
curl -s -u admin:changeme \
  "http://observability-vm.lan:3000/api/datasources/proxy/uid/loki/loki/api/v1/label/node/values" \
  | python -c "import sys,json; print(sorted(json.load(sys.stdin).get('data',[])))"

# Loki: log volume per node in the last 15 min (catches sparse but present nodes)
curl -s -u admin:changeme --get \
  "http://observability-vm.lan:3000/api/datasources/proxy/uid/loki/loki/api/v1/query" \
  --data-urlencode 'query=sum by (node) (count_over_time({node=~".+"}[15m]))' \
  | python -c "import sys,json; [print(f'  {x[\"metric\"][\"node\"]:22} {x[\"value\"][1]:>6}') for x in sorted(json.load(sys.stdin)['data']['result'], key=lambda r: r['metric']['node'])]"

# Prometheus: scrape targets + per-target health
curl -s -u admin:changeme \
  "http://observability-vm.lan:3000/api/datasources/proxy/uid/prometheus/api/v1/targets?state=any" \
  | python -c "import sys,json; [print(f'  {t[\"labels\"][\"job\"]:20} {t[\"health\"]}') for t in json.load(sys.stdin)['data']['activeTargets']]"

Common patterns when a VM "isn't shipping":

See [[feedback-telemetry-label-check]] for the durable habit and rationale.

Troubleshooting decision tree

LiteLLM — full telemetry wiring

The LiteLLM proxy (on litellm-gateway) is wired into all three pillars. Each is independent and answers different questions:

How traces flow

LiteLLM pushes OTLP HTTP to Tempo via an OTel callback:

# configs/litellm/settings.yaml (local-inference-infrastructure)
litellm_settings:
  callbacks: ["prometheus", "otel", "lmstudio_context_hook.proxy_handler_instance"]

# In the container env, written by provision-litellm-gateway → templates/litellm.env.j2
OTEL_EXPORTER_OTLP_ENDPOINT=http://observability-vm.lan:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

The env block in the template is conditionally rendered ONLY when observability_server_ip is passed as an Ansible var. Without it, the otel callback can still be in the callbacks list — but the OTel SDK retries forever to reach an empty endpoint, delaying LiteLLM startup by 30+ seconds. If you ever take observability-vm down without removing otel from the callbacks list, expect slow LiteLLM restarts.

How Prom scrape flows (auth dance)

LiteLLM's /metrics is gated behind the master key. To scrape:

1. Master key lives in Infisical at /litellm/LITELLM_MASTER_KEY in the shared homelab project (duplicated from the litellm-specific project — same value, easier than cross-project access).
2. deploy-observability.yml loads it as LITELLM_MASTER_KEY env var via the secrets-action.
3. A workflow step SSHes into observability-vm, writes the token to ~/services/observability/secrets/litellm-token (mode 0440, owned by 65534 = Prom). Secret piped via stdin → sudo tee so it never appears in argv or logs. Parent dir must be 0711 so Prom UID can traverse (see homelab-bootstrap skill).
4. docker-compose.yml bind-mounts ./secrets:/etc/prometheus/secrets:ro.
5. prometheus.yml references via:

   authorization:
     type: Bearer
     credentials_file: /etc/prometheus/secrets/litellm-token
   

Useful litellm_* metric names

There are 54 series total. The most informative for "is the proxy healthy" dashboards:

Key labels to slice by:

• model — the actually-routed model (e.g. claude-sonnet-4-6, local-reasoning)
• requested_model — what the client asked for (e.g. anthropic_auto)
• api_provider — anthropic, openai, etc.

Proxmox host + per-VM metrics — pve-exporter sidecar

Proxmox doesn't expose Prometheus metrics natively. We run prompve/prometheus-pve-exporter as a compose sidecar on observability-vm. Prometheus scrapes the exporter, the exporter queries the PVE API.

Pieces

# docker-compose.yml
pve-exporter:
  image: prompve/prometheus-pve-exporter:latest
  restart: unless-stopped
  ports:
    - "127.0.0.1:9221:9221"   # internal only — Prometheus on same host
  env_file:
    - ./secrets/pve.env

# prometheus.yml — bridge scrape
- job_name: 'pve'
  metrics_path: /pve
  params:
    target: ['pve.lan']         # which PVE host to query
  static_configs:
    - targets: ['pve-exporter:9221']   # docker network alias
      labels:
        node: 'pve'

# secrets/pve.env  (mode 0400, owned by ubuntu; written by deploy workflow)
PVE_USER=terraform@pam
PVE_TOKEN_NAME=ci-token
PVE_TOKEN_VALUE=<UUID>
PVE_VERIFY_SSL=false

The deploy workflow loads /proxmox/PROXMOX_VE_API_TOKEN from Infisical (format user@realm!tokenid=uuid), splits it into the three env fields, and writes the env file via sudo tee (NOT plain tee) — see [[feedback-redeployable-secret-files]].

Useful exposed metrics

To get per-VM CPU labeled by name (not vmid), join on pve_guest_info:

pve_cpu_usage_ratio{id=~"qemu/.+"} * on(id) group_left(name) pve_guest_info

Permission setup on Proxmox

Tokens need Sys.Audit on / (PVEAuditor covers it). Token privsep trap — see proxmox-api skill's "API token permissions" section. The short version: if privsep=0, you must grant on the user, not the token. The wrong-place grant produces an identical 403 forever.

Adding a second PVE host

The exporter is multi-tenant. Add another scrape with a different target param and a different node label — same sidecar serves both. Set PVE_USER / PVE_TOKEN_NAME / PVE_TOKEN_VALUE for each endpoint in pve.env if hosts have different tokens (see exporter docs for the <endpoint_name>_user=… syntax).

Technitium DNS metrics — technitium-exporter sidecar

Same compose-sidecar pattern as pve-exporter. Image: ghcr.io/guycalledseven/technitium-dns-prometheus-exporter:latest — combines window-based snapshots (top domains, top clients, blocked, DHCP) with v15+ lifetime counters. Listens on :9105/metrics.

Pieces

# docker-compose.yml
technitium-exporter:
  image: ghcr.io/guycalledseven/technitium-dns-prometheus-exporter:latest
  restart: unless-stopped
  ports:
    - "127.0.0.1:9105:9105"
  env_file:
    - ./secrets/technitium.env

# prometheus.yml
- job_name: 'technitium'
  static_configs:
    - targets: ['technitium-exporter:9105']
      labels:
        node: 'dns-vm'

# secrets/technitium.env  (sudo-tee'd, 0400 owned by ubuntu)
TECHNITIUM_BASE_URL=http://dns-vm.lan:5380
TECHNITIUM_TOKEN=<from Infisical /technitium/TECHNITIUM_TOKEN>
TECHNITIUM_VERIFY_SSL=false
TECHNITIUM_STATS_RANGE=LastHour
TECHNITIUM_TOP_LIMIT=25
SERVER_LABEL=dns-vm

The token at Infisical /technitium/TECHNITIUM_TOKEN is the same one the kenske/technitium Terraform provider and the technitium-api skill use — admin scope, more than enough for the exporter's read-only API calls. Best practice would be a dedicated read-only token; not done here for expedience and because the homelab's blast radius is tiny.

Useful exposed metrics

Cache hit ratio:

sum(technitium_dns_queries_window{category="cached"})
/ sum(technitium_dns_queries_window{category="all"})

Block ratio: same shape, swap cached → blocked.

Companion Grafana dashboard

Dashboard id=24555 ("Technitium DNS Exporter") on grafana.com is the companion to this exporter — 33 panels, broad coverage. Drop in via the standard recipe from "Adding a Grafana dashboard (declarative)" above (curl download → ${DS_PROMETHEUS} rewrite → id=null → also drop the __inputs / __requires blocks, which are UI-import scaffolding and trip Grafana's file provisioner if left in).

Our custom companion is dns-network-activity.json — 19 panels, slim "morning coffee" view (totals + cache/block ratios + protocol mix + top blocked + top clients + DHCP). Built via the Python generator pattern.

UFW reminder

If the deploy probe reports technitium down with connection refused on first deploy, dns-vm's UFW is blocking observability-vm on :5380. Open it with the standard pattern from homelab-platform's base-vm.yml / dns playbook — see [[feedback-prometheus-scrape-through-ufw]]. (On our current setup the port was already LAN-open because the user hits the UI from their workstation, so no UFW change was needed.)

Building dashboards — Python generator pattern

The Local Inference Overview dashboard has 27 panels across 6 rows. Hand-writing that JSON is hostile (Grafana's schema has a lot of defaults and per-panel quirks). The pattern that worked:

1. Write a Python script under .dashboard-gen.py (gitignored — it's a build tool, not source) that emits the final JSON.
2. Use helper functions per panel type (stat(), timeseries(), logs(), trace_table()) that take the minimum-bothering args and fill in sane defaults.
3. Pin data source UIDs ({"type":"prometheus","uid":"prometheus"}) so the JSON works against our provisioning.
4. Always set j["id"] = None — Grafana assigns one on load; reusing a stale id causes "Failed to import" on re-provisioning.
5. Layout: 24-col grid. Stat panels w=4-6, timeseries w=8, logs w=12, wide tables w=24. Each row's y = sum of previous heights.
6. Bump version in the dashboard meta when shipping changes (helps with Grafana's change-tracking).

To regenerate: python .dashboard-gen.py from the repo root, commit the JSON output, delete the script. Reify the script next time you need to extend the dashboard (it's a small file, easy to recreate).

The grafana-folder-provider is set to updateIntervalSeconds: 30 and allowUiUpdates: true — quick iteration loop: edit script → regenerate → push → 30s later it's live.

TraceQL metrics in Grafana panels

Tempo data source query types in Grafana panels:

• "queryType": "traceqlSearch" with a filters array → table panel of trace search results
• "queryType": "metrics" with a TraceQL query string ending in | rate() or | quantile_over_time(...) → timeseries panel

TraceQL metrics use nanoseconds for duration. Use "unit": "ns" on latency timeseries panels so Grafana auto-formats.

Related

• [[project-homelab-observability]] — what's deployed
• [[project-homelab-dns]] — Technitium serves the .lan zone this stack uses
• [[feedback-dns-record-cname-pattern]] — observability.lan is a CNAME, not a pinned A
• homelab-bootstrap skill — broader gotchas (localhost-in-container, docker_container handler, systemd-resolved dropin, force-recreate, container UIDs, workflow_dispatch 422, heredoc-in-YAML)
• homelab-dashboard skill — dashboard tiles link to the LGTM endpoints
• Upstream docs: grafana.com/oss for LGTM components