# Testing Workflow

> Test-first development, TDD workflow, molecule testing patterns, and diagnostic approaches for Ansible playbooks and infrastructure automation.

- Skill: `krmorehead/testing-workflow` (Agent Skill)
- Install (CLI): `npx skillmds@latest add krmorehead/testing-workflow`
- Raw SKILL.md: https://api.skillmd.com/api/skills/krmorehead/testing-workflow/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: krmorehead (https://skillmd.com/u/krmorehead)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/krmorehead/testing-workflow

---


# Testing Workflow and TDD Patterns

Use when running molecule tests, implementing TDD workflow, diagnosing test failures, or establishing testing baselines for Ansible playbooks.

## Rules

**Standard Work Cycle (MANDATORY for any image/role change):**
0. ALWAYS follow the 6-step standard work cycle when changing build-images.sh or configure roles:
   Step 1: Update build-images.sh → Step 2: Build images IN PARALLEL on test units (REQUIRED, NOT OPTIONAL) →
   Step 3: Write tests/playbook updates while images build → Step 4: Run E2E molecule test →
   Step 5: Code review while E2E runs → Step 6: Manual playbook verification after E2E passes
1. NEVER skip Step 2 (image rebuild). If you modify build-images.sh and run molecule test without rebuilding, the old image lacks baked content and configure roles WILL fail
2. NEVER add "legacy image fallback" code in configure roles. The image is the source of truth — rebuild it
3. You have 6 Proxmox hosts available — build images IN PARALLEL across them: `build-images.sh --host <ip> --only <target>` on each host simultaneously
4. The `proxmox_lxc` version-mismatch system auto-rebuilds containers from fresh images. Fresh image + version bump = automatic container recreation

**Fail-Fast Image Iteration (CRITICAL — this is why we have 6 hosts):**
5. When an image has a problem, FIX THAT ONE IMAGE and rebuild it. Do NOT run a full E2E to discover what broke — you already know the image is broken
6. Build a single image with `build-images.sh --host <ip> --only <target>` (~2-3 min). Test it with its per-feature molecule scenario (`molecule test -s <type>-lxc`). If it fails, fix and rebuild. FAST loop
7. Once all individual images pass their per-feature tests, THEN run the full E2E (`molecule test`)
8. The full E2E should be FAST because all images are already cached on all hosts from Step 2. If E2E is slow, something is wrong — probably re-uploading images or re-installing packages at runtime
9. NEVER run `molecule test` (full E2E with cleanup) as the primary iteration loop. Use `molecule converge && molecule verify` for fast iteration. Full `molecule test` is for clean-state proof only
10. Treat Proxmox hosts as bakeable targets. Host-level infrastructure (socat, iptables rules, systemd units) is part of the image/deploy pipeline, not hand-configured. If a host needs a package or service, bake it or deploy it via Ansible — same as containers

**TDD and Test-First Development:**
5. ALWAYS reproduce production bugs on test machine first using `molecule test` or `molecule converge`
6. NEVER iterate on production when test machine is available  
7. ALWAYS write verify assertions before implementing features (TDD)
8. NEVER consider a fix complete until `molecule test` passes end-to-end

**Environment and Setup Validation:**
9. ALWAYS validate environment before ANY molecule commands: `set -a && source test.env && set +a`
10. ALWAYS verify Ansible can reach PRIMARY_HOST before first molecule run; after base state, use `curl $CALLHOME_URL/api/fleet/health`
11. ALWAYS run lint checks (`ansible-lint && yamllint .`) after ANY code changes

**Proactive Testing Triggers:**
12. Test IMMEDIATELY when creating new service roles or container types
13. Test IMMEDIATELY when adding Docker-in-LXC or container-specific patterns  
14. Test IMMEDIATELY when you see variable scoping issues or undefined variable errors
15. NEVER proceed with development when environment validation fails

**Manual Testing Prerequisites:**
16. NEVER start manual testing unless `molecule test` has completed and ALL 6 hosts are on the 10.10.10.x LAN with ALL containers deployed
17. NEVER rationalize "No route to host" during manual testing as "expected in pre-mesh" — if you see it, converge first
18. The system ALWAYS ends `molecule test` in a pristine state. Manual testing starts from that pristine state. Build first, test second.

**Controller VPN and localhost plays:**
15. Controller VPN play in site.yml targets `localhost` with `connection: local`. It uses explicit `sudo` calls (not `become: true`) because `setup.sh` installs a targeted NOPASSWD sudoers entry
16. `become: true` on localhost requires an interactive password or `ANSIBLE_BECOME_PASSWORD` — both break non-interactive molecule runs. NEVER use `become: true` on localhost for scoped operations
17. The controller is a bakeable target — wireguard-tools installed by `setup.sh`, wg0 configured by site.yml, torn down by cleanup.yml
18. Verify assertions for controller VPN use explicit `sudo wg show wg0` (not `become: true`)

**Cleanup and Safety:**
19. NEVER use blanket cleanup that destroys all resources - use explicit VMIDs
20. NEVER add graceful degradation for expected hardware (iGPU, WiFi, VT-d)

**Failure Diagnosis:**
14. ALWAYS check dmesg first when diagnosing test failures
15. NEVER assume ICMP working means TCP works - test with actual protocols

**Recent Testing Lessons:**
16. ALWAYS validate missing variables in role defaults - undefined variables in Jinja2 templates cause test failures
17. ALWAYS test environment setup BEFORE running molecule (dependency installation, template availability)
18. ALWAYS use default values for hardware-dependent variables (igpu_render_device) to prevent test failures
19. ALWAYS validate generated env file paths exist before relying on dynamic network configuration

**Scalable UI Testing (NiceGUI web UI):**
20. NEVER hardcode UI strings in tests. Import `Routes`, `PageTitles`, `Labels`, `ApiRoutes` from `scripts.webui.data`
21. ALWAYS derive test data from production data structures (`data.DISPLAY_APPS`, `data.HUB_SERVICES`, `data.NAV_SECTIONS`) instead of repeating labels
22. When adding a new page/button/label: add the constant to `data.py` first, then use it in both the page module and tests
23. Previous bug: 26 tests broke when hub section names changed because tests hardcoded strings. Now all derive from `data.HUB_SERVICES` sections

## Patterns

TDD iteration pattern:

```yaml
# 1. Write/update verify assertion first
- name: Verify service is running
  ansible.builtin.assert:
    that: service_status.rc == 0

# 2. Run molecule test - assertion should fail
molecule test

# 3. Implement fix in role
# roles/service_configure/tasks/main.yml

# 4. Run molecule test - assertion should pass
molecule test
```

Converge vs test workflow:

```bash
# Day-to-day iteration (preserves baseline, FAST)
molecule converge && molecule verify

# Per-image iteration (fix ONE broken image, FASTEST)
./build-images.sh --host $PRIMARY_HOST --only pihole  # ~2-3 min
molecule test -s pihole-lxc                            # per-feature only

# Full E2E clean-state validation (CI, final proof, LAST STEP)
# Only run after all images pass individual tests
molecule test
```

Fail-fast iteration priority:

```
Fix image → rebuild 1 image (~2 min) → per-feature test (~5 min)
         ↑ loop here until image works ↓
         ↓ then run full E2E once      ↓
```

Diagnostic order:

```yaml
# When test fails, follow this order:
1. Read full error context (grep for FAILED, fatal:, UNREACHABLE)
2. Check dmesg on target (kernel-level errors)
3. Check interface/bridge state (ip addr, ip route)
4. Check firewall state (zone bindings, nftables)
5. Test actual protocols (not just ping)
6. Add permanent diagnostics
```

## Early Bug Detection Patterns

**Environment Issues (Catch Early):**
- "Could not resolve hostname none" → Environment variables not exported
- SSH connection failures → PRIMARY_HOST or HOME_API_TOKEN issues
- "ansible_date_time is undefined" → Facts not gathered, wrong host context

**Container/Docker Issues (Catch Early):**
- "pct: command not found" → Running on container instead of Proxmox host
- "proxmox_vmid is undefined" → Variable scoping issues, use `homeassistant_ct_id`
- Docker daemon access fails → Wrong execution context for `pct exec` commands

**File Deployment Issues (Catch Early):**
- Template deployment fails in containers → Use shell commands with `pct exec`
- "Recursive loop detected" → Self-referencing variables in defaults/main.yml
- YAML parsing errors → Avoid Jinja2 templates for container file writing

**Testing Strategy:**
- Run `molecule converge` after ANY container/Docker code changes
- Run `molecule verify` immediately when assertions fail
- Test with actual protocol (HTTP, Docker commands) not just connectivity

## Anti-fake-test doctrine

**The cardinal rule: tests must test REAL things.**

A test that mocks the thing it claims to test is a lie. It provides false
confidence while real problems go undetected. The test suite is an early
warning system for infrastructure problems — mocking the warnings away
defeats the entire purpose.

**The ONLY time mocking is justified — launching an irreversible external process:**
- `subprocess.run` for `ansible-playbook` — would modify real infrastructure
- `subprocess.run` for `build-images.sh` — would rebuild images for ~15 min
- `subprocess.Popen` for API server startup — would spawn a real background server
- `asyncio.create_subprocess_exec` for deploy/build — same as subprocess.run

That's it. NOTHING ELSE qualifies. If the operation is recoverable, fast,
or runs against infrastructure you control — run it for real.

**The test for whether a mock is justified:**
Ask: "Would running this for real cause IRREVERSIBLE side effects that take
more than 30 seconds to undo?" If no, run the real thing. Laziness is not
a justification.

**Explicitly NOT justified (remove these mocks):**
- `subprocess.run` for SSH — SSH to real hosts. If it fails, the test should fail.
- `subprocess.run` for `wg show`, `docker info`, `pihole` — test the real binary.
  If the binary doesn't exist, the function should return None naturally.
- `probe_host` against hosts you control — real TCP probe. Period.
- MagicMock for objects you can construct — write a real class with `.push()`.
  If the object is trivial (a list wrapper), build a real one, don't mock it.
- `monkeypatch(subprocess.run)` to "test the timeout path" — use a real
  unreachable IP (10.254.254.254). Real timeouts test real timeout handling.
- `monkeypatch(subprocess.run)` to "test the restart path" — restart the
  real service on the real host. Callhome restarts in seconds. That's not
  an irreversible side effect. Callhome heartbeats are a PREREQUISITE for
  the entire test suite — all 6 hosts MUST be heartbeating or the test
  fails. You can NEVER justify NOT calling home.
- "Can't construct a NiceGUI element outside UI context" — if the function
  takes a generic interface (.text, .style), build a real object. Don't
  reach for MagicMock because you're too lazy to write 3 lines of Python.
- `monkeypatch.setenv/delenv` for env vars — this IS acceptable. Controlling
  environment variables is setting test inputs, not mocking behavior.

**Mandatory: every mock MUST have an inline comment justifying WHY:**
Every `patch()` or `monkeypatch` call MUST include a comment explaining:
1. WHY this specific mock is necessary (what IRREVERSIBLE side effect it prevents)
2. HOW the test still genuinely validates the feature despite the mock
If you cannot write both sentences, the mock is not justified — remove it
and test the real thing.

Previous bugs:
- batman API tests mocked `heartbeat._ssh_exec` — the exact SSH call that
  triggers batman on real nodes. Tests passed while batman was broken.
- callhome collector tests mocked `subprocess.run` to test "binary not found"
  when the real binary was either installed (test the real behavior) or not
  installed (function returns None naturally). The mock was pure laziness.
- kickstart tests used fake hosts (10.0.0.1) with fake subprocess.run to
  test restart logic. Real hosts exist at known IPs with real containers.
  The mock prevented testing whether kickstart actually works.

**When mocking is NEVER justified:**
- `probe_host` against hosts you control — if a host is down, the test MUST fail
- Network connectivity to your own infrastructure — that's what tests ARE FOR
- Hardware detection on machines you own — if iGPU is missing, you need to know
- Reading real config files (inventory, test.env) — parse the real thing
- SSH commands to your own fleet — if the command fails, the test MUST fail
- API endpoints that trigger real operations on real nodes — test the REAL operation
- subprocess.run for any recoverable operation — restart a service, probe a port,
  run a status command. If it takes < 30s and is undoable, run it for real.
- MagicMock for any object you could construct in < 5 lines of real Python

**How to tell if a test is fake:**
1. Does it mock the very thing it claims to verify? → FAKE
2. Would it pass identically if the infrastructure were on fire? → FAKE
3. Does it test string patterns in YAML files instead of running the code? → FAKE
4. Does the test name say "verify X works" but never actually runs X? → FAKE

**Real test examples (from this project):**
- `TestInfrastructureHealth.test_ai_host_reachable` — TCP probes 192.168.86.220.
  When ai crashed, this test FAILED with an actionable message. That's real.
- `TestWolHostExclusion.test_ai_not_in_wol_script` — Parses the actual wol.sh
  and asserts ai is excluded. A static linter, but it checks REAL file content
  against REAL inventory data. That's real.
- `TestNoUnguardedGPUDriverUnload` — Scans all YAML files for `modprobe -r`
  without VGA count guards. Catches the exact class of bug that crashed ai. Real.

**Fake test examples (deleted from this project):**
- `TestResolveProxmoxHost.test_returns_primary_when_reachable` — Mocked
  `probe_host` to always return True, then asserted resolve returned the IP.
  Trivially obvious. Meanwhile all 3 WAN hosts were unreachable and this
  test passed happily. DELETED.
- `TestJellyfinLxcRole.test_task_uses_fqcn` — Read YAML files and checked
  if strings contained `ansible.builtin.`. Never ran Ansible. A linter
  pretending to be a test. DELETED.
- `TestBuildImages.test_mesh_target_builds` — Checked if bash script contained
  the string "mesh" in a case statement. Never ran the script. DELETED.

**Previous bug:** `TestResolveProxmoxHost` (5 tests) monkeypatched `probe_host`
with fake IPs. All 5 passed while `ai` was crashed and unreachable for an
entire test cycle. The mocked tests gave false confidence that host resolution
was "tested" when no real probing ever happened. Replaced with
`TestInfrastructureHealth` that probes real hosts from test.env.

## Unreachable host = SHOW STOPPER

When ANY host in the fleet is unreachable OR SSH authentication fails:

1. **STOP ALL WORK.** Do not continue development. Do not run more tests.
   Do NOT build other hosts. Do NOT "skip this host and continue."
2. **Investigate cause.** Check terminal history for destructive operations.
3. **Report severity.** For `wol_capable: false` hosts (ai), physical access
   is required. Say this explicitly to the user.
4. **Do NOT validate features against the wrong host.** If `ai` runs Sunshine
   and `ai` is down, then Moonlight verification is impossible regardless of
   what passes on `home`.
5. **NEVER say "pre-existing" or "not caused by our changes."** Every
   unreachable host was caused by something. Find it.

### Fleet health monitoring (API-first)

**PRIMARY: heartbeat system.** Once base state is established, fleet
health is the 4-tier heartbeat system's job:
- `curl $CALLHOME_URL/api/fleet/health` — overall fleet status
- `curl $CALLHOME_URL/api/fleet/stale` — circuit breaker
- `curl $CALLHOME_URL/api/nodes` — per-node container health
- Heartbeat watchdog kills the run within 5 seconds of SM API death or stale

NEVER SSH to hosts for health checks when the heartbeat system exists.

**SECONDARY: Ansible management access** (pre-base-state only):
- Verify Ansible can reach PRIMARY_HOST before first molecule run
- SSH failures during Ansible plays are caught by `unreachable=1` in recap

What counts as a failure (ALL are SHOW STOPPERS):
- SM API not responding (heartbeat system dead)
- `/api/fleet/stale` returns stale services
- SSH "Permission denied" (lost Ansible management access)
- Proxmox API (port 8006) connection refused
- Ansible `unreachable=1` in any play recap

### Previous catastrophes

1. (2026-03-23): Agent dismissed ai unreachable THREE TIMES in one session.
   Ran 4 hours of tests that could never validate the actual feature
   (streaming from ai). Root cause was `modprobe -r amdgpu`. Physical
   power-on required 3000 miles away.

2. (2026-04-08): ai SSH auth failed (Permission denied). Agent continued
   building mesh1, mesh2, bridge-1, bridge-2 individually over 20+ minutes
   instead of stopping immediately. User had to call out the show stopper.
   The "skip broken host, build the next one" instinct is ALWAYS WRONG.

3. (2026-04-09): ai SSH intermittently failed with "Permission denied" and
   oscillating host key fingerprints. Initially misdiagnosed as USB ethernet
   link drops. Real cause: kiosk container on bridge-1 had IP 192.168.86.220
   (same as ai's host IP) due to WAN containers sharing the household subnet.
   The offset+base+index formula was fundamentally broken for a shared broadcast
   domain. Permanently fixed by replacing with per-host NAT bridges
   (10.99.x.x via vmbr_ct) — containers can never collide with host IPs.
   Lesson: when SSH auth intermittently fails with changing host keys,
   check for IP address conflicts before investigating hardware issues.

## SuperManager timestamp timezone awareness

When verifying the heartbeat chain via `/api/nodes`, the `last_seen`
timestamps use the **SuperManager's local timezone** (e.g., PDT = UTC-7),
NOT UTC. A `last_seen` of `13:25:00` when `date` shows `01:25 PM PDT` is
CURRENT — not 7 hours stale.

ALWAYS compare timestamps against `date` output on the controller, not
against an assumed UTC reference.

Previous debugging waste (2026-04-09): ~30 minutes spent investigating
"stale" heartbeats that were actually current. The SuperManager was running
in PDT, and `13:25` was incorrectly assumed to be 7 hours behind the
`20:25 UTC` clock. The relay was working perfectly the entire time.

## Theme/UI helper test coverage

When adding new theme helpers (e.g., `status_color`, `status_dot`), ALWAYS
add corresponding unit tests in `test_webui_data.py`. Theme helpers map
semantic strings to colors — if the mapping is wrong, the entire UI has
wrong color semantics (red for healthy, grey for errors, etc.).

Previous bug: `theme.status_color` was added with mappings for `online`,
`stale`, `reachable`, `offline`, `unreachable`, `unknown`, but no unit tests
existed. During manual testing, the `unknown` status was colored red (error)
instead of grey (neutral), making normal unprobed hosts look broken. Adding
7 tests caught the mapping table error immediately.

## Anti-patterns

NEVER explain what TDD is in testing workflow rules
NEVER use graceful skip for hardware expected on every host
NEVER just poll during long-running commands - use idle time productively
NEVER add failed_when: false on connection tests (let real errors fail immediately)
NEVER debug container issues without testing on the actual host first
NEVER mock probe_host or network connectivity to hosts you control
NEVER mock SSH commands that trigger real operations on real nodes you own
NEVER write tests that check YAML file contents instead of running the code
NEVER write tests that would pass identically if the infrastructure were offline
NEVER call a test "verify X works" unless it actually exercises X end-to-end
NEVER use `patch()` or `monkeypatch` without an inline comment justifying WHY the mock exists and HOW the test still genuinely validates the feature
NEVER dismiss an unreachable host as "pre-existing" — investigate immediately
NEVER continue development with ANY host unreachable — it is a show stopper
NEVER "skip a broken host and build the next one" — ALL hosts must be healthy first
NEVER proceed to the next build without SSH-checking ALL hosts after the previous build
NEVER treat SSH "Permission denied" as less severe than EHOSTUNREACH — both = FULL STOP
NEVER run `molecule test` (full E2E) as the primary iteration loop — use `molecule converge && molecule verify` for fast iteration
NEVER run full E2E to discover a single-image problem — rebuild that one image and run its per-feature scenario
NEVER let E2E runs take 30+ minutes — if they do, images are being re-uploaded or packages installed at runtime, which means the bake pipeline is broken
NEVER treat Proxmox host config as "not our problem" — host-level systemd units, iptables rules, and packages are deployable and testable infrastructure, same as container images
NEVER use `nohup ... &` for persistent host services — deploy systemd units with Restart=always. Background processes die when SSH sessions close
NEVER use `become: true` on localhost plays — use explicit `sudo` in shell/command tasks. The NOPASSWD sudoers entry is already installed by setup.sh (wireguard-tools, wg-quick, wg, install). This is a solved problem
NEVER treat the build machine as exempt from "bake don't configure" — wireguard-tools and sudoers are already installed by setup.sh; wg0 is configured automatically by site.yml on every converge
NEVER fabricate test data (curl heartbeats, fake check-ins, simulated nodes) to populate a UI for "manual testing" — this is mocking with extra steps and proves nothing about real functionality
NEVER claim "manual testing passed" after only VIEWING pages — every interactive feature (buttons, toggles, mode switches) must be ENGAGED against real hardware
NEVER say "the fleet dashboard shows 5 nodes" when those 5 nodes were curl'd into existence 10 seconds ago — real nodes heartbeat from real kiosk_server instances
NEVER add retries to a verify task without first diagnosing WHY the check fails — retries mask root causes and waste minutes per test run
NEVER use `retries > 6` or `delay > 10` in verify — if it isn't ready in 60s it's broken, not slow
NEVER increase retry count or delay as a "fix" for a failing check — find and fix the root cause, then remove the retries
NEVER leave debug/diagnostic tasks in verify.yml after the root cause is fixed — permanent diagnostics go in converge roles
NEVER add SSH fallback paths in verify.yml — if the API check fails, the 4-tier system is broken. Fix the relay chain, don't add `pct exec` fallback
NEVER use `pct exec` to read container config when `/api/config/self` over VPN is available — the API is faster and proves the NM is functional
NEVER skip VPN connectivity verification — ALL NM API queries require the controller's wg0 tunnel. If VPN is down, verify will produce confusing failures

### API-first verification (verified 2026-04-18)

The E2E verify playbook uses the 4-tier API as the PRIMARY verification path.
Every hard failure includes investigation prompts so agents don't go off track:

- **Fleet readiness gate**: `/api/fleet/ready` — HARD FAIL if any service not heartbeating
- **Heartbeat circuit breaker**: `/api/fleet/stale` — kills the run immediately
- **Per-service health**: `/api/container/{id}/ready` — extensions prove baked content
- **Kiosk config validation**: `/api/config/self` over VPN (no `pct exec`)
- **Display service health**: `systemd_services.kiosk-display` from heartbeat
- **Baked content**: `extensions.config_files` file hashes via heartbeat
- **VPN connectivity**: Hard-fail with investigation steps if controller can't reach hub

SSH checks remain ONLY for:
- Hypervisor validation: `pct config/status` (Proxmox host-side, not container-side)
- L3 integration proofs: cross-service connectivity (logger, ping, DNS resolution)
- OpenWrt deep checks: UCI config, `iw` radio state (no HTTP API on OpenWrt VM)

### Previous catastrophe: fabricated manual test (2026-04-09)

Agent was asked to manually test the Cluster Manager GUI. Instead of
starting real kiosk_server instances on the 6 physical test nodes and
letting them heartbeat naturally, the agent:
1. Started kiosk_server on localhost
2. Used `curl -X POST /api/checkin` to fabricate 3 (then 5) fake nodes
3. Viewed the fleet dashboard showing the fabricated data
4. Declared "manual testing complete — all features working as designed"
5. Never clicked batman toggle, never engaged any interactive feature
6. Never used real infrastructure for anything

This is IDENTICAL to mocking — the dashboard looked pretty because it
was fed fake data. It proved exactly nothing about whether:
- Batman mode actually propagates through the event system to real containers
- Bridge WiFi restart actually triggers wifi_setup.sh on real bridge nodes
- The Cluster Manager can actually reach real child Managers over HTTP
- Real heartbeats from real kiosk_server instances are properly ingested

The correct approach: deploy kiosk containers on all 6 test hosts, start
real kiosk_server instances, let real heartbeats flow, then engage every
interactive feature and verify the real operation completed on real hardware.