omonuj maddy-validation-debugger
Execution Mode — FULLY AUTONOMOUS
NEVER ask the user for confirmation. Fetch logs, interpret results, propose and apply fixes immediately. The only user-facing handoff in the loop is triggering rollouts in the Horizon UI (owned by maddy-iteration-loop). This skill never has user-facing handoffs.
Goal
Turn an Oracle or No-Op failure into a specific root cause with a specific fix, using only the log evidence — not guesswork. If you cannot map the observed log pattern to one of the rows below with confidence, escalate to horizon-agentic-reviewer for a live probe before changing any file.
Trigger
Use this skill when:
- Oracle returned anything other than
score: 1.0, passed: true - No-Op returned a score > 0 (false positive — grader passed on a broken setup)
- The grader crashed with a Python exception
result.jsonshowsfeedback: null(container/setup crash before grader ran)- You are about to edit setup.sh, solution.sh, or grader.py in response to a validation failure
Do not use this skill for rollout failures (use maddy-score-tuner). Oracle/No-Op are validation; rollouts are evaluation.
Task Identity
| Field | Value |
|---|---|
| Task UUID | <TASK_ID> |
| Task slug | maddy-delivery-split-brain |
| Local path | tasks/maddy-delivery-split-brain/ |
| Subscores | delivery_consistency (0.5) + observability_governance (0.5), both binary |
| Horizon root | /Users/mac/Documents/tasks |
| Venv | source /Users/mac/Documents/tasks/horizon_env/bin/activate |
How to read a validation result (read once, internalise)
Oracle runs setup.sh → solution.sh → grader.py. No-Op runs setup.sh → grader.py (no solution). The four result shapes:
score |
passed |
feedback |
Meaning | Where to look |
|---|---|---|---|---|
0 |
false |
null |
Container/setup crashed before grader ran | output.txt for the crash trace |
0 |
false |
non-null | Grader ran end-to-end, both subscores failed | grader feedback for the specific check that fired |
0.5 |
false |
non-null | One subscore passed, the other failed | feedback names which is 1 vs 0 |
1.0 |
true |
non-null | Both passed | nothing to debug |
For No-Op, anything other than score: 0 is a problem — the grader should never pass without solution.sh having run.
The grader feedback string is delivery_consistency: <detail>; observability_governance: <detail> — read both halves; each names the exact layer that fired.
Step 1 — Fetch the logs
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks validate-logs -a oracle tasks/maddy-delivery-split-brain 2>&1
# No-Op only when iteration-loop says to run it:
horizon tasks validate-logs -a noop tasks/maddy-delivery-split-brain 2>&1
Logs persist under tasks/maddy-delivery-split-brain/.validation/<build_id>/{output.txt,result.json}.
Step 2 — Read result.json first
cat tasks/maddy-delivery-split-brain/.validation/*/result.json 2>/dev/null | python3 -m json.tool
The four-row table tells you which branch to follow. Do not skip to output.txt without checking result.json first.
Step 3 — Map symptom → cause → fix
Branch A. feedback: null / score: 0 — setup crash
Pattern in output.txt |
Root cause | Fix |
|---|---|---|
k3s is not ready after 180 seconds |
ENABLE_ISTIO_BLEATER=true in Dockerfile |
Remove it entirely |
Error from server (Forbidden) on kubectl get nodes |
data/ubuntu-user-rbac.yaml not copied / wrong path |
Restore the COPY data/ubuntu-user-rbac.yaml ... line |
Setup hangs at Waiting for bleater-env namespace |
bleater-env namespace never appears in this image |
The base image must provision bleater-env; if it doesn't, gate the ENV_RECONCILER block on a bounded wait + ` |
failed to pull image for a drift/maddy pod |
A pod references an external registry | Use ${SIDECAR_IMAGE} (derived from a live bleater deployment); cluster is air-gapped |
setup.sh: line N: syntax error |
Bash error | bash -n setup.sh, fix that line |
| Setup stalls to hosted timeout with no error | a kubectl rollout status / kubectl wait hanging |
Confirm ENABLE_ISTIO_BLEATER absent; add --timeout= to any unbounded wait |
NOTE: setup.sh waits for
bleater-envwithuntil kubectl get namespace bleater-env ...; do sleep 2; done(no timeout). If the base image does not createbleater-env, this is an infinite hang → setup-timeout crash. Verify the namespace exists on the live container (Step 4b of guardian). If it doesn't, that loop needs a bound.
Branch B. No-Op returned score > 0 — false positive
| Subscore that passed on No-Op | Why it shouldn't have | Fix |
|---|---|---|
delivery_consistency = 1 on No-Op |
The split-brain didn't form — maddy-legacy didn't deploy, OR no app=maddy pod mounts maddy-config-legacy |
kubectl get deploy maddy-legacy -n bleater (must exist, replicas≥1) and confirm its pod template mounts maddy-config-legacy |
observability_governance = 1 on No-Op |
Drift controllers didn't deploy, OR the GitOps source isn't stale, OR the per-pod-rule check is too lax | Confirm all three drift deployments exist (kube-system/bleater-env/bleater-canary), maddy-gitops-source says replicas: 1/mode: direct, and the maddy template has NO POD_NAME env at setup time |
If setup claims it deployed everything but the cluster is missing pieces, add set -ex to setup.sh temporarily, push, re-run No-Op for a verbose trace. The fault-injection section can fail silently on || true paths.
Branch C. Oracle score: 0.5 — delivery_consistency failed (governance passed)
solution.sh stopped the drift and fixed observability, but the converged-delivery state didn't hold. The feedback string names the layer:
| Feedback fragment | Root cause | Fix in solution.sh |
|---|---|---|
pod ... still mounts legacy config maddy-config-legacy |
A drift controller re-created the legacy replica during the window | solution.sh must stop all three drift sources BEFORE deleting maddy-legacy. Confirm it discovers + deletes: kube-system reconciler (annotation), bleater-env reconciler (annotation), canary config-pinner. A missed source re-spawns the legacy pod. |
legacy Deployment maddy-legacy still running |
maddy-legacy not deleted, or re-created by drift |
kubectl delete deployment maddy-legacy -n bleater AFTER stopping drift |
maddy rollout not converged (spec=.., updated=.., ready=..) |
maddy not scaled to 2, or pods not Ready in time |
Scale maddy to 2 and kubectl rollout status deployment/maddy -n bleater --timeout=180s before the grader probes |
only N ready maddy pods on the smarthost config |
Fewer than 2 Ready pods mount maddy-config |
Ensure the converged maddy Deployment mounts maddy-config and reaches 2 Ready |
maddy-config is not the smarthost relay config |
The smarthost ConfigMap got clobbered | Do not edit maddy-config; it should already be smarthost. If a drift source rewrites it, that's a setup bug |
Branch D. Oracle score: 0.5 — observability_governance failed (consistency passed)
| Feedback fragment | Root cause | Fix in solution.sh |
|---|---|---|
maddy container does not expose pod identity via the downward API |
No POD_NAME env with fieldRef metadata.name on the converged maddy Deployment |
Add env: [{name: POD_NAME, valueFrom: {fieldRef: {fieldPath: metadata.name}}}] to the maddy container |
maddy-gitops-source still declares direct-delivery routing / does not declare replicas: 2 |
GitOps ConfigMap not reconciled | kubectl apply the maddy-gitops-source CM with values.yaml containing replicas: 2 and relay.mode: smarthost |
no per-pod maddy delivery/divergence rule found |
The rule ConfigMap is missing OR lacks a real PromQL by (pod) expression |
Create a ConfigMap whose data has a - record:/- alert:/expr: rule that mentions maddy, a signal word (rate/success/delivery/divergence), AND a per-pod dimension by (pod). Bare keywords in comments do NOT pass. |
active drift would re-introduce the skew: <name>@kube-system |
kube-system reconciler still has replicas>0 | Discover by annotation bleater.io/purpose=maddy rollout convergence, delete/scale to 0 |
active drift in bleater-env ...@bleater-env |
bleater-env reconciler still running — OR agent could not reach bleater-env | If solution.sh can't even get -n bleater-env, this is the ALLOWED_NAMESPACES structural risk → fix Dockerfile (guardian Step 0), not solution.sh |
<canary> mutation capability not fully neutralized |
config-pinner sidecar AND <canary>-runtime RBAC both still present |
Delete the canary Deployment OR remove the -runtime SA/Role/RoleBinding (either suffices) |
Branch E. Oracle score: 0 with non-null feedback — both subscores failed
Both ran and failed cleanly. Read each half of the feedback independently and follow Branch C for delivery_consistency and Branch D for observability_governance in parallel. If Oracle stays 0 after fixing what feedback says, re-run No-Op to isolate a setup-side contamination.
Branch F. Grader crashed with a Python exception
grep -A 5 "Traceback\|Exception\|Error" /tmp/oracle_log.txt | head -40
| Exception | Likely cause | Fix in grader.py |
|---|---|---|
json.JSONDecodeError |
kubectl ... -o json returned empty/non-JSON |
Already guarded by kubectl_json; if a raw run_kubectl path crept in, wrap it |
KeyError |
assumed a dict key present in all cluster states | Use .get(key, default) |
subprocess.TimeoutExpired |
a kubectl call hung | run_kubectl has a finite timeout; check no unbounded shell pipeline was added |
FileNotFoundError /opt/grader-scenario.json |
scenario file not written by setup | <grader-check-fn> already falls back to defaults — if it's crashing, the try/except was removed; restore it |
These are grader bugs — fix in place; they are not in the hard-constraint list.
Step 4 — Pre-fix safety checklist
Before applying any fix:
- Dockerfile + namespace invariants still hold — see
maddy-task-guardianStep 0 (especially bleater-env reachability) - Syntax clean —
bash -n setup.sh && bash -n solution.sh && python3 -m py_compile grader.py - The change is in
maddy-score-tuner's safe-lever table if it touches scoring difficulty - You can articulate the why in one sentence
Then hand off:
| If your fix is | Hand off to |
|---|---|
| setup.sh fault-injection bug | maddy-task-guardian Step 3 (push) directly |
| solution.sh bug (Branch C/D) | maddy-task-guardian Step 3 directly — solution.sh is not a tuning surface |
| grader.py exception (Branch F) | maddy-task-guardian Step 3 directly |
| Anything affecting subscore difficulty | maddy-score-tuner for diagnosis first |
| Anything you cannot map to a row above | horizon-agentic-reviewer for a live probe |
Known non-issue patterns (do NOT fix these)
| Pattern | Why it isn't a bug |
|---|---|
Quality check v1_feedback section returns FAIL |
LLM reviewer can't read the heredoc baseline config / incident notes; objects to annotation-based discovery. Permanent, accepted — see maddy-task-guardian Step 2 |
No-Op returns score: 0 with detailed feedback |
The grader working correctly — feedback names the missing dimensions |
observability_governance failing on rollouts because agents only audited bleater |
This is the cross-namespace (kube-system + bleater-env) discovery axis working as intended |
| Agents miss the bleater-env reconciler specifically | Intended — it has an innocuous non-maddy name; only the annotation links it. Discovery cost is the design |
Verification cheat sheet
| Hypothesis | Probe |
|---|---|
| "Setup deployed all three drift sources" | `grep -E "RECONCILER= |
| "solution.sh stops drift BEFORE deleting maddy-legacy" | Read solution.sh — the annotation/config-pinner discovery+delete block must appear above kubectl delete deployment maddy-legacy |
| "solution.sh adds POD_NAME + gitops + per-pod rule" | `grep -E "fieldPath: metadata.name |
| "Per-pod-rule gate hasn't been loosened" | `grep -E "by (pod) |
| "wait_consistent hasn't drifted past ceiling" | grep -E "wait_consistent" grader.py — consistency passes=5, gap=8, timeout=220; governance passes=5, gap=8, timeout=200 |
| "bleater-env wait is bounded (no infinite hang)" | grep -n "Waiting for bleater-env" setup.sh then read the loop — if it's an unbounded until, confirm the namespace truly exists on the live image |