Homelab Service Debugging
Why this exists
A homelab with 6-7+ VMs/LXCs has too many layers to debug from scratch each time. "The service is down" could mean the host, the guest OS, the process, a dependency, or the network — and each layer has a different fix. Proxmox showing the guest as "running" only confirms the hypervisor layer; it says nothing about the four layers above it. This skill is the fixed order to check them in, so triage starts at the cheapest check and only goes deeper once each layer is confirmed innocent.
Procedure
Work top-down. Stop and fix as soon as a layer explains the symptom — don't keep going once you've found the cause.
1. Confirm what "down" actually means
- Connection refused, timeout, or an HTTP error (502/503)? These point to different layers: refused means nothing's listening, timeout means a network/firewall block, 502/503 means the process is up but a proxy or dependency behind it isn't.
- Can't-load-in-browser vs. API/CLI both failing narrows whether it's the app layer or something lower.
2. Host layer (Proxmox)
- Check actual guest status, not just "started" — a VM/LXC can show as running while its init process crashed.
pvesh get /nodes/<node>/qemu or /lxc/<vmid>/status/current, or the Proxmox UI's Status tab.
- Check the Proxmox host's own resource pressure (CPU/RAM/disk), not just the guest's — a starved host can silently stall guests without marking them down.
- Any recent host reboot, migration, or snapshot restore that might not have restarted a service inside the guest?
3. Guest OS reachability
ping the guest IP first.- Then try SSH. Ping-ok-but-SSH-fails narrows the problem to an OS-level service or network-service issue, not a fully dead guest.
4. Process / container layer
- Is the actual process or container running?
systemctl status <service> for a bare service, docker compose ps for a compose stack.
- Read the real error before guessing:
journalctl -u <service> -e or docker compose logs <service> — "it's down" is a symptom, the log line is the cause.
5. Resource exhaustion inside the guest
- Disk full:
df -h, especially for anything with a database or verbose logs.
- OOM kill:
dmesg | grep -i oom, or check if Docker itself OOM-killed a container (docker inspect <container> → OOMKilled).
6. Dependency layer
- If the service has a database/cache in the same stack, is that container/process actually up? A web app container showing "running" with a dead Postgres/Redis dependency is one of the most common false "it's just down" reports.
- Reverse proxy / port mapping correct? A container can be healthy internally but unreachable because of a proxy misconfig, or still inside a slow-start healthcheck window.
7. Network layer
- Right VLAN, and no firewall rule blocking it — especially worth checking after any recent VLAN/firewall change.
- DNS resolving to the expected IP, if accessed by hostname rather than IP directly.
Worked example: Netbox on LXC dradis
Concrete run-through of the same order, using this homelab's actual Netbox deployment (LXC dradis, VMID 106, Proxmox node enterprise, Docker Compose at /opt/netbox):
- Symptom was "page won't load" — check whether it's a timeout (network/host) or connection refused (nothing listening) before touching anything.
- Host layer: confirm
dradis shows running in Proxmox, and separately check node enterprise isn't under memory/disk pressure.
- Guest reachability:
ping 192.168.30.27, then SSH into dradis.
- Process layer:
docker compose ps in /opt/netbox — Netbox is a multi-container stack (app, Postgres, Redis, nginx), so "the container" is plural here; check all of them, not just the one named netbox.
- Resource exhaustion: this LXC previously hit swap exhaustion at 2GB RAM during a device-type import (now provisioned at 4GB) —
free -h and dmesg | grep -i oom are worth checking first given that history.
- Dependency layer: Netbox's Postgres/Redis containers being down produces a running-but-broken app container — check those before assuming the app itself is the problem.
- Network/proxy: the compose stack's nginx healthcheck has a
start_period: 120s — a restart can look "down" for up to two minutes before it's actually failed.
1---2name: homelab-service-debugging3description: Use whenever a self-hosted homelab service, VM, or LXC is reported down, unreachable, timing out, or behaving unexpectedly — even if Proxmox shows the host as running. Walks a fixed host-to-network layer order (Proxmox host → guest OS → process/container → dependency → network) instead of guessing where to start. Trigger on phrases like "X isn't running", "can't reach X", "X is down but the VM is on", or any "where do I even start debugging this" homelab outage.4---56# Homelab Service Debugging78## Why this exists910A homelab with 6-7+ VMs/LXCs has too many layers to debug from scratch each time. "The service is down" could mean the host, the guest OS, the process, a dependency, or the network — and each layer has a different fix. Proxmox showing the guest as "running" only confirms the hypervisor layer; it says nothing about the four layers above it. This skill is the fixed order to check them in, so triage starts at the cheapest check and only goes deeper once each layer is confirmed innocent.1112## Procedure1314Work top-down. Stop and fix as soon as a layer explains the symptom — don't keep going once you've found the cause.1516### 1. Confirm what "down" actually means1718- Connection refused, timeout, or an HTTP error (502/503)? These point to different layers: refused means nothing's listening, timeout means a network/firewall block, 502/503 means the process is up but a proxy or dependency behind it isn't.19- Can't-load-in-browser vs. API/CLI both failing narrows whether it's the app layer or something lower.2021### 2. Host layer (Proxmox)2223- Check actual guest status, not just "started" — a VM/LXC can show as running while its init process crashed. `pvesh get /nodes/<node>/qemu` or `/lxc/<vmid>/status/current`, or the Proxmox UI's Status tab.24- Check the Proxmox **host's** own resource pressure (CPU/RAM/disk), not just the guest's — a starved host can silently stall guests without marking them down.25- Any recent host reboot, migration, or snapshot restore that might not have restarted a service inside the guest?2627### 3. Guest OS reachability2829- `ping` the guest IP first.30- Then try SSH. Ping-ok-but-SSH-fails narrows the problem to an OS-level service or network-service issue, not a fully dead guest.3132### 4. Process / container layer3334- Is the actual process or container running? `systemctl status <service>` for a bare service, `docker compose ps` for a compose stack.35- Read the real error before guessing: `journalctl -u <service> -e` or `docker compose logs <service>` — "it's down" is a symptom, the log line is the cause.3637### 5. Resource exhaustion inside the guest3839- Disk full: `df -h`, especially for anything with a database or verbose logs.40- OOM kill: `dmesg | grep -i oom`, or check if Docker itself OOM-killed a container (`docker inspect <container>` → `OOMKilled`).4142### 6. Dependency layer4344- If the service has a database/cache in the same stack, is *that* container/process actually up? A web app container showing "running" with a dead Postgres/Redis dependency is one of the most common false "it's just down" reports.45- Reverse proxy / port mapping correct? A container can be healthy internally but unreachable because of a proxy misconfig, or still inside a slow-start healthcheck window.4647### 7. Network layer4849- Right VLAN, and no firewall rule blocking it — especially worth checking after any recent VLAN/firewall change.50- DNS resolving to the expected IP, if accessed by hostname rather than IP directly.5152## Worked example: Netbox on LXC `dradis`5354Concrete run-through of the same order, using this homelab's actual Netbox deployment (LXC `dradis`, VMID 106, Proxmox node `enterprise`, Docker Compose at `/opt/netbox`):55561. Symptom was "page won't load" — check whether it's a timeout (network/host) or connection refused (nothing listening) before touching anything.572. Host layer: confirm `dradis` shows running in Proxmox, and separately check node `enterprise` isn't under memory/disk pressure.583. Guest reachability: `ping 192.168.30.27`, then SSH into `dradis`.594. Process layer: `docker compose ps` in `/opt/netbox` — Netbox is a multi-container stack (app, Postgres, Redis, nginx), so "the container" is plural here; check all of them, not just the one named `netbox`.605. Resource exhaustion: this LXC previously hit swap exhaustion at 2GB RAM during a device-type import (now provisioned at 4GB) — `free -h` and `dmesg | grep -i oom` are worth checking first given that history.616. Dependency layer: Netbox's Postgres/Redis containers being down produces a running-but-broken app container — check those before assuming the app itself is the problem.627. Network/proxy: the compose stack's nginx healthcheck has a `start_period: 120s` — a restart can look "down" for up to two minutes before it's actually failed.
katelovescode