# Maddy Validation Debugger

> Fetches and interprets validation logs (No-Op and Oracle) for the maddy-delivery-split-brain task (<TASK_ID>). Maps observed log patterns to root causes and the smallest safe fix. Use after a validation fails to understand why it failed before editing anything.

- Skill: `omonuj/maddy-validation-debugger` (Agent Skill)
- Install (CLI): `npx skillmds@latest add omonuj/maddy-validation-debugger`
- Raw SKILL.md: https://api.skillmd.com/api/skills/omonuj/maddy-validation-debugger/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: omonuj (https://skillmd.com/u/omonuj)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/omonuj/maddy-validation-debugger

---


# maddy-validation-debugger

## Execution Mode — FULLY AUTONOMOUS

**NEVER ask the user for confirmation.** Fetch logs, interpret results, propose and apply fixes immediately. The only user-facing handoff in the loop is triggering rollouts in the Horizon UI (owned by `maddy-iteration-loop`). This skill never has user-facing handoffs.

## Goal

Turn an Oracle or No-Op failure into a specific root cause with a specific fix, using only the log evidence — not guesswork. If you cannot map the observed log pattern to one of the rows below with confidence, escalate to `horizon-agentic-reviewer` for a live probe before changing any file.

## Trigger

Use this skill when:
- Oracle returned anything other than `score: 1.0, passed: true`
- No-Op returned a score > 0 (false positive — grader passed on a broken setup)
- The grader crashed with a Python exception
- `result.json` shows `feedback: null` (container/setup crash before grader ran)
- You are about to edit setup.sh, solution.sh, or grader.py in response to a validation failure

Do **not** use this skill for rollout failures (use `maddy-score-tuner`). Oracle/No-Op are validation; rollouts are evaluation.

## Task Identity

| Field | Value |
|---|---|
| **Task UUID** | `<TASK_ID>` |
| **Task slug** | `maddy-delivery-split-brain` |
| **Local path** | `tasks/maddy-delivery-split-brain/` |
| **Subscores** | `delivery_consistency` (0.5) + `observability_governance` (0.5), both binary |
| **Horizon root** | `/Users/mac/Documents/tasks` |
| **Venv** | `source /Users/mac/Documents/tasks/horizon_env/bin/activate` |

## How to read a validation result (read once, internalise)

Oracle runs `setup.sh → solution.sh → grader.py`. No-Op runs `setup.sh → grader.py` (no solution). The four result shapes:

| `score` | `passed` | `feedback` | Meaning | Where to look |
|---|---|---|---|---|
| `0` | `false` | `null` | Container/setup crashed before grader ran | `output.txt` for the crash trace |
| `0` | `false` | non-null | Grader ran end-to-end, both subscores failed | grader feedback for the specific check that fired |
| `0.5` | `false` | non-null | One subscore passed, the other failed | feedback names which is 1 vs 0 |
| `1.0` | `true` | non-null | Both passed | nothing to debug |

For No-Op, **anything other than `score: 0` is a problem** — the grader should never pass without solution.sh having run.

The grader `feedback` string is `delivery_consistency: <detail>; observability_governance: <detail>` — read both halves; each names the exact layer that fired.

---

## Step 1 — Fetch the logs

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks validate-logs -a oracle tasks/maddy-delivery-split-brain 2>&1
# No-Op only when iteration-loop says to run it:
horizon tasks validate-logs -a noop tasks/maddy-delivery-split-brain 2>&1
```

Logs persist under `tasks/maddy-delivery-split-brain/.validation/<build_id>/{output.txt,result.json}`.

## Step 2 — Read `result.json` first

```bash
cat tasks/maddy-delivery-split-brain/.validation/*/result.json 2>/dev/null | python3 -m json.tool
```

The four-row table tells you which branch to follow. **Do not skip to `output.txt` without checking `result.json` first.**

## Step 3 — Map symptom → cause → fix

### Branch A. `feedback: null` / `score: 0` — setup crash

| Pattern in `output.txt` | Root cause | Fix |
|---|---|---|
| `k3s is not ready after 180 seconds` | `ENABLE_ISTIO_BLEATER=true` in Dockerfile | Remove it entirely |
| `Error from server (Forbidden)` on `kubectl get nodes` | `data/ubuntu-user-rbac.yaml` not copied / wrong path | Restore the `COPY data/ubuntu-user-rbac.yaml ...` line |
| Setup hangs at `Waiting for bleater-env namespace` | `bleater-env` namespace never appears in this image | The base image must provision `bleater-env`; if it doesn't, gate the ENV_RECONCILER block on a bounded wait + `|| { echo skip; }` instead of an infinite `until` loop |
| `failed to pull image` for a drift/maddy pod | A pod references an external registry | Use `${SIDECAR_IMAGE}` (derived from a live bleater deployment); cluster is air-gapped |
| `setup.sh: line N: syntax error` | Bash error | `bash -n setup.sh`, fix that line |
| Setup stalls to hosted timeout with no error | a `kubectl rollout status` / `kubectl wait` hanging | Confirm `ENABLE_ISTIO_BLEATER` absent; add `--timeout=` to any unbounded wait |

> NOTE: setup.sh waits for `bleater-env` with `until kubectl get namespace bleater-env ...; do sleep 2; done` (no timeout). If the base image does not create `bleater-env`, this is an **infinite hang** → setup-timeout crash. Verify the namespace exists on the live container (Step 4b of guardian). If it doesn't, that loop needs a bound.

### Branch B. No-Op returned `score > 0` — false positive

| Subscore that passed on No-Op | Why it shouldn't have | Fix |
|---|---|---|
| `delivery_consistency = 1` on No-Op | The split-brain didn't form — `maddy-legacy` didn't deploy, OR no `app=maddy` pod mounts `maddy-config-legacy` | `kubectl get deploy maddy-legacy -n bleater` (must exist, replicas≥1) and confirm its pod template mounts `maddy-config-legacy` |
| `observability_governance = 1` on No-Op | Drift controllers didn't deploy, OR the GitOps source isn't stale, OR the per-pod-rule check is too lax | Confirm all three drift deployments exist (kube-system/bleater-env/bleater-canary), `maddy-gitops-source` says `replicas: 1`/`mode: direct`, and the maddy template has NO `POD_NAME` env at setup time |

If setup claims it deployed everything but the cluster is missing pieces, add `set -ex` to setup.sh temporarily, push, re-run No-Op for a verbose trace. The fault-injection section can fail silently on `|| true` paths.

### Branch C. Oracle `score: 0.5` — `delivery_consistency` failed (governance passed)

solution.sh stopped the drift and fixed observability, but the converged-delivery state didn't hold. The feedback string names the layer:

| Feedback fragment | Root cause | Fix in solution.sh |
|---|---|---|
| `pod ... still mounts legacy config maddy-config-legacy` | A drift controller re-created the legacy replica during the window | solution.sh must stop **all three** drift sources BEFORE deleting `maddy-legacy`. Confirm it discovers + deletes: kube-system reconciler (annotation), bleater-env reconciler (annotation), canary `config-pinner`. A missed source re-spawns the legacy pod. |
| `legacy Deployment maddy-legacy still running` | `maddy-legacy` not deleted, or re-created by drift | `kubectl delete deployment maddy-legacy -n bleater` AFTER stopping drift |
| `maddy rollout not converged (spec=.., updated=.., ready=..)` | `maddy` not scaled to 2, or pods not Ready in time | Scale `maddy` to `2` and `kubectl rollout status deployment/maddy -n bleater --timeout=180s` before the grader probes |
| `only N ready maddy pods on the smarthost config` | Fewer than 2 Ready pods mount `maddy-config` | Ensure the converged `maddy` Deployment mounts `maddy-config` and reaches 2 Ready |
| `maddy-config is not the smarthost relay config` | The smarthost ConfigMap got clobbered | Do not edit `maddy-config`; it should already be smarthost. If a drift source rewrites it, that's a setup bug |

### Branch D. Oracle `score: 0.5` — `observability_governance` failed (consistency passed)

| Feedback fragment | Root cause | Fix in solution.sh |
|---|---|---|
| `maddy container does not expose pod identity via the downward API` | No `POD_NAME` env with `fieldRef metadata.name` on the converged `maddy` Deployment | Add `env: [{name: POD_NAME, valueFrom: {fieldRef: {fieldPath: metadata.name}}}]` to the `maddy` container |
| `maddy-gitops-source still declares direct-delivery routing` / `does not declare replicas: 2` | GitOps ConfigMap not reconciled | `kubectl apply` the `maddy-gitops-source` CM with `values.yaml` containing `replicas: 2` and `relay.mode: smarthost` |
| `no per-pod maddy delivery/divergence rule found` | The rule ConfigMap is missing OR lacks a real PromQL `by (pod)` expression | Create a ConfigMap whose data has a `- record:`/`- alert:`/`expr:` rule that mentions `maddy`, a signal word (`rate`/`success`/`delivery`/`divergence`), AND a per-pod dimension `by (pod)`. Bare keywords in comments do NOT pass. |
| `active drift would re-introduce the skew: <name>@kube-system` | kube-system reconciler still has replicas>0 | Discover by annotation `bleater.io/purpose=maddy rollout convergence`, delete/scale to 0 |
| `active drift in bleater-env ...@bleater-env` | bleater-env reconciler still running — OR **agent could not reach bleater-env** | If solution.sh can't even `get -n bleater-env`, this is the `ALLOWED_NAMESPACES` structural risk → fix Dockerfile (guardian Step 0), not solution.sh |
| `<canary> mutation capability not fully neutralized` | `config-pinner` sidecar AND `<canary>-runtime` RBAC both still present | Delete the canary Deployment OR remove the `-runtime` SA/Role/RoleBinding (either suffices) |

### Branch E. Oracle `score: 0` with non-null feedback — both subscores failed

Both ran and failed cleanly. Read each half of the feedback independently and follow Branch C for `delivery_consistency` and Branch D for `observability_governance` in parallel. If Oracle stays `0` after fixing what feedback says, re-run No-Op to isolate a setup-side contamination.

### Branch F. Grader crashed with a Python exception

```bash
grep -A 5 "Traceback\|Exception\|Error" /tmp/oracle_log.txt | head -40
```

| Exception | Likely cause | Fix in grader.py |
|---|---|---|
| `json.JSONDecodeError` | `kubectl ... -o json` returned empty/non-JSON | Already guarded by `kubectl_json`; if a raw `run_kubectl` path crept in, wrap it |
| `KeyError` | assumed a dict key present in all cluster states | Use `.get(key, default)` |
| `subprocess.TimeoutExpired` | a kubectl call hung | `run_kubectl` has a finite timeout; check no unbounded shell pipeline was added |
| `FileNotFoundError /opt/grader-scenario.json` | scenario file not written by setup | `<grader-check-fn>` already falls back to defaults — if it's crashing, the try/except was removed; restore it |

These are grader bugs — fix in place; they are not in the hard-constraint list.

---

## Step 4 — Pre-fix safety checklist

Before applying any fix:
1. **Dockerfile + namespace invariants still hold** — see `maddy-task-guardian` Step 0 (especially bleater-env reachability)
2. **Syntax clean** — `bash -n setup.sh && bash -n solution.sh && python3 -m py_compile grader.py`
3. **The change is in `maddy-score-tuner`'s safe-lever table** if it touches scoring difficulty
4. **You can articulate the why** in one sentence

Then hand off:

| If your fix is | Hand off to |
|---|---|
| setup.sh fault-injection bug | `maddy-task-guardian` Step 3 (push) directly |
| solution.sh bug (Branch C/D) | `maddy-task-guardian` Step 3 directly — solution.sh is not a tuning surface |
| grader.py exception (Branch F) | `maddy-task-guardian` Step 3 directly |
| Anything affecting subscore difficulty | `maddy-score-tuner` for diagnosis first |
| Anything you cannot map to a row above | `horizon-agentic-reviewer` for a live probe |

---

## Known non-issue patterns (do NOT fix these)

| Pattern | Why it isn't a bug |
|---|---|
| Quality check `v1_feedback` section returns FAIL | LLM reviewer can't read the heredoc baseline config / incident notes; objects to annotation-based discovery. Permanent, accepted — see `maddy-task-guardian` Step 2 |
| No-Op returns `score: 0` with detailed feedback | The grader working correctly — feedback names the missing dimensions |
| `observability_governance` failing on rollouts because agents only audited `bleater` | This is the cross-namespace (kube-system + bleater-env) discovery axis working as intended |
| Agents miss the bleater-env reconciler specifically | Intended — it has an innocuous non-maddy name; only the annotation links it. Discovery cost is the design |

---

## Verification cheat sheet

| Hypothesis | Probe |
|---|---|
| "Setup deployed all three drift sources" | `grep -E "RECONCILER=|ENV_RECONCILER=|MADDY_CANARY=" /tmp/oracle_log.txt` — all three scenario echoes should appear |
| "solution.sh stops drift BEFORE deleting maddy-legacy" | Read solution.sh — the annotation/config-pinner discovery+delete block must appear above `kubectl delete deployment maddy-legacy` |
| "solution.sh adds POD_NAME + gitops + per-pod rule" | `grep -E "fieldPath: metadata.name|by \(pod\)|replicas: 2" solution.sh` — all three must appear |
| "Per-pod-rule gate hasn't been loosened" | `grep -E "by \(pod\)|- record:|- alert:|expr:" grader.py` — these must remain in the Layer-3 check |
| "wait_consistent hasn't drifted past ceiling" | `grep -E "wait_consistent" grader.py` — consistency `passes=5, gap=8, timeout=220`; governance `passes=5, gap=8, timeout=200` |
| "bleater-env wait is bounded (no infinite hang)" | `grep -n "Waiting for bleater-env" setup.sh` then read the loop — if it's an unbounded `until`, confirm the namespace truly exists on the live image |