# Early Validation Patterns

> Proactive validation patterns to catch issues early, prevent debugging blind, and establish testing habits before problems escalate.

- Skill: `krmorehead/early-validation-patterns` (Agent Skill)
- Install (CLI): `npx skillmds@latest add krmorehead/early-validation-patterns`
- Raw SKILL.md: https://api.skillmd.com/api/skills/krmorehead/early-validation-patterns/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: krmorehead (https://skillmd.com/u/krmorehead)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/krmorehead/early-validation-patterns

---


# Early Validation and Proactive Testing Patterns

Use when starting new projects, implementing new service types, or when you want to prevent issues before they become major problems.

## Critical Validation Sequence (MANDATORY)

**Before ANY development work:**
```bash
set -a && source test.env && set +a
echo "PRIMARY_HOST: $PRIMARY_HOST"
echo "AI_HOST: $AI_HOST"
echo "MESH_2_HOST: $MESH_2_HOST"

# If base state is already established, use the heartbeat system:
curl -sf http://localhost:$WEBUI_PORT/api/fleet/health && echo "Fleet OK"
curl -sf http://localhost:$WEBUI_PORT/api/nodes | python3 -m json.tool

# If base state is NOT established (first run), verify Ansible access:
ansible all -m ping
```

**If the SM API is not responding or any service is stale → FULL STOP.**
The heartbeat system IS the health check. If `/api/fleet/health` fails,
the heart stopped — fix the SM, don't SSH around it.

**If ANY host is unreachable → FULL STOP. Investigate immediately.**
Do NOT continue development with a degraded fleet. An unreachable host is a
5-alarm emergency, especially for `ai` (USB ethernet, no WoL, no remote
recovery — requires physical power-on 3000 miles away).

**NEVER dismiss an unreachable host as "pre-existing."** Every unreachable
host was caused by something. Find the cause. If automation crashed it,
that is a code bug that MUST be fixed before any other work continues.

Previous catastrophe: `ai` went unreachable during a Moonlight implementation
session. The agent dismissed it as "pre-existing" THREE TIMES across 4 hours
of testing, running converge and verify cycles that could never validate the
actual streaming feature (ai IS the Sunshine/Doom server). The root cause was
`modprobe -r amdgpu` from an earlier sunshine-vm cleanup — the exact known
crash vector. The dismissal wasted the entire session and left ai requiring
physical power-on with no remote recovery path.

## Proactive Testing Triggers

**Test IMMEDIATELY when you:**
- Create a new service role (`<service>_lxc`, `<service>_configure`)
- Add Docker-in-LXC or container functionality
- Use `pct exec` commands
- Encounter "undefined variable" errors
- See "Could not resolve hostname none"
- Add any Ansible template modules for containers
- Modify variable scoping or host context

**Previous lesson**: We created `homeassistant_configure` but didn't test until much later. We should have run `molecule converge` immediately after role creation.

## Environment Variable Patterns

**Proper environment export:**
```bash
# CORRECT: Exports all variables
set -a && source test.env && set +a

# WRONG: Variables not available to child processes
source test.env
```

**Molecule environment handling:**
- Molecule uses `${VAR}` syntax in molecule.yml
- Variables MUST be exported for molecule to access them
- Test with `echo $HOME_API_TOKEN` before running molecule

## Code Quality Validation (Run After Every Change)

```bash
# After any Ansible code changes:
source .venv/bin/activate
ansible-lint roles/<service>_*/ 
yamllint roles/<service>_*/ molecule/<scenario>/
```

**Previous lesson**: We fixed linting issues at the end instead of catching them early. Run lint checks continuously.

## Skill Loading Patterns

**Load skills proactively when you recognize the domain:**
- LXC/Container work → `lxc-container-patterns`
- Docker-in-LXC → `lxc-container-patterns` + `docker-related` (when available)
- Proxmox operations → `proxmox-safety-rules`
- New Ansible development → `ansible-conventions`
- Testing work → `molecule-testing`

**Previous lesson**: We struggled with Docker-in-LXC patterns because we didn't load `lxc-container-patterns` until late. Load relevant skills immediately when you recognize the domain.

## Early Bug Detection

**Common early warning signs:**
- "Could not resolve hostname none" → Environment not exported
- "pct: command not found" → Wrong execution context
- "proxmox_vmid is undefined" → Variable scoping issues
- Template failures in containers → Use shell + pct exec instead

**When you see these → Test immediately instead of debugging blind.**

## Test-First Development Habits

1. **Write verify assertion FIRST** → Then implement the feature
2. **Test after EVERY significant change** → Don't batch multiple untested changes
3. **Reproduce on test machine FIRST** → Never debug production directly
4. **Run full test cycle BEFORE declaring success** → `molecule test` passes end-to-end

## Common Failure Prevention

| Early Symptom | Likely Cause | Prevention |
|---------------|--------------|------------|
| "hostname none" | Environment not exported | Always use `set -a && source test.env && set +a` |
| UNREACHABLE hosts | Host or heartbeat connectivity | Check `curl $CALLHOME_URL/api/fleet/health` first; if pre-base-state, use `ansible home -m ping` |
| Container template failures | Missing nesting=1 or cgroup | Load `lxc-container-patterns` early |
| Docker access fails | Wrong execution context | Use `service_nodes` host, not container group |
| SSH auth intermittently fails + host key oscillates | IP address conflict (container on another host has same IP) | Check `pct config` on all hosts for IP conflicts before investigating hardware |
| SSH "Permission denied" on freshly installed host | Another device/container answering on the same IP | Run `arp -n` and verify WAN containers use NAT bridge (10.99.x.x), not the household subnet |

## Development Workflow Integration

**At the start of ANY new service implementation:**
1. Load relevant skills proactively
2. Run environment validation
3. Write first verify assertion
4. Test that assertion fails (TDD)
5. Implement minimal feature
6. Test that assertion passes
7. Continue with small iterations

**Result**: Catch issues in minutes instead of hours of debugging.

## Manual testing means REAL testing

"Manual testing" is NOT viewing a UI populated with fabricated data. It is
exercising every interactive feature against real infrastructure and verifying
the real operation completed on real hardware.

**NEVER fabricate data to make a UI look correct during manual testing:**
- NEVER `curl -X POST /api/checkin` to create fake nodes
- NEVER script fake heartbeats to populate a dashboard
- NEVER claim "testing passed" after only viewing pages without clicking anything

**ALWAYS test against real infrastructure:**
- Start real services on real hosts (deploy with `molecule converge` if needed)
- Let real heartbeats flow from real containers
- Click every button, toggle, and action on the page
- Verify the real outcome via the fleet API (`/api/fleet/ready`, `/api/container/{id}/ready`) and host-side `pct config`/`qm config` for hypervisor checks

Fabricated test data is mocking with extra steps. It proves the UI can render
JSON — not that the system works.

Previous catastrophe (2026-04-09): Agent curled fake heartbeats into the
Cluster Manager, viewed the dashboard, never clicked batman toggle or any
other feature, and declared "manual testing complete." Every feature was
untested. The dashboard looked pretty because it was fed fabricated data.