# Maddy Iteration Loop

> Orchestrator skill that runs the full push → Oracle → live red-team → rollout → tune cycle for the maddy-delivery-split-brain task (<TASK_ID>) until Oracle scores 1.0 and rollout avg < <TARGET_MEAN> with both subscores varying. Delegates to maddy-task-guardian, maddy-validation-debugger, maddy-score-tuner, and horizon-agentic-reviewer at the right phases. The agent itself executes every step except triggering the rollout batch.

- Skill: `omonuj/maddy-iteration-loop` (Agent Skill)
- Install (CLI): `npx skillmds@latest add omonuj/maddy-iteration-loop`
- Raw SKILL.md: https://api.skillmd.com/api/skills/omonuj/maddy-iteration-loop/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: omonuj (https://skillmd.com/u/omonuj)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/omonuj/maddy-iteration-loop

---


# maddy-iteration-loop

## Execution Mode — FULLY AUTONOMOUS

**NEVER call `ask_user` or pause for confirmation at any point.** Run every phase, every bash command, and every fix without asking permission. If a decision has two valid paths, pick the safer one and proceed. The user triggered this skill precisely to avoid being prompted — do not interrupt them.

## Goal

Drive the maddy-delivery-split-brain task from "needs work" to "shippable" without human intervention except for triggering the rollout batch in the Horizon UI. The loop is **done** when all exit conditions hold; until then, every cycle picks the right specialist skill (validation-debugger, score-tuner, horizon-agentic-reviewer) for the symptom and applies the smallest safe fix.

This is the front-door skill for "iterate this task to a passing state." The other three maddy-* skills are pure capabilities; this skill orchestrates them.

## Trigger

Use this skill when asked to:
- "Iterate the maddy task until it passes"
- "Push and tune until avg < <TARGET_MEAN>"
- "Run the full loop on maddy-delivery-split-brain"
- Anything implying repeated push → validate → analyze → fix cycles

For single-step requests (just push, just analyze rollouts, just debug a validation), invoke the relevant specialist skill directly.

## Task Identity

| Field | Value |
|---|---|
| **Task UUID** | `<TASK_ID>` |
| **Task slug** | `maddy-delivery-split-brain` |
| **Local path** | `tasks/maddy-delivery-split-brain/` |
| **Horizon root** | `/Users/mac/Documents/tasks` |
| **Venv** | `source /Users/mac/Documents/tasks/horizon_env/bin/activate` |

---

## Who does what

**The agent (you) executes every step except 2.5c.** Never tell the user to run push, validate, or any CLI command.

| Step | Who |
|---|---|
| Edit files (setup.sh, grader.py, solution.sh, task.yaml, Dockerfile) | **Agent** |
| Pre-push syntax + anatomy + quality checks | **Agent** |
| Push (`horizon tasks push`) | **Agent** |
| Oracle validation | **Agent** |
| Live red-team | **Agent** (invokes `horizon-agentic-reviewer`) |
| Pull + analyze rollouts | **Agent** |
| Diagnose tuning / validation fixes | **Agent** (invokes `maddy-score-tuner` / `maddy-validation-debugger`) |
| **Trigger the eval batch in the Horizon UI** | **User** (the only manual step) |

---

## Exit conditions

The loop is DONE when **all** hold on the same version:
1. Oracle: `passed: true, score: 1.0`, both subscores `=1`
2. Live red-team: no BLOCKING findings — **especially bleater-env reachability** (see `maddy-task-guardian` Step 4b)
3. Rollout avg: `< <TARGET_MEAN>`
4. `delivery_consistency`: varies (both 0 and 1 appear)
5. `observability_governance`: varies (both 0 and 1 appear)
6. The two subscores are **not perfectly correlated** (at least some `0.5` rollouts appear, not only 0.0 and 1.0)
7. Local quality: 18 substantive checks pass (the `v1_feedback` section may FAIL as accepted noise)

Anything short of all seven → loop continues.

## Loop limit

**Do not exceed 5 push cycles without human review.** After 5 cycles with no measurable progress on a specific failure mode, stop and summarize what was tried and what's still failing.

## No-Op default

**Do NOT run No-Op as part of the loop. Oracle only.** Oracle exercises the same setup.sh path. Run No-Op manually only when Oracle returns `feedback: null` / `score: 0` with no grader detail (setup-crash isolation).

---

## The loop

```
START
  ▼
PHASE 0: Pre-push checks ─────────────────────────────────────────
  • Dockerfile + namespace invariants (maddy-task-guardian Step 0)
    — incl. bleater-env reachability risk
  • bash -n setup.sh && bash -n solution.sh && py_compile grader
  • horizon check-anatomy (must pass clean)
  • horizon check-quality (18 substantive pass; v1_feedback may FAIL)
  If anything else fails → fix → re-run PHASE 0
  ▼
PHASE 1: Push ────────────────────────────────────────────────────
  horizon tasks push → record version NNN
  ▼
PHASE 2: Oracle validation ───────────────────────────────────────
  → 1.0 + both subscores=1 → PHASE 2.5
  → anything else → maddy-validation-debugger → fix → PHASE 0
  ▼
PHASE 2.5: Live red-team ──────────────────────────────────────────
  invoke horizon-agentic-reviewer on the UUID
  → no BLOCKING finding → ask user to trigger rollout batch
  → BLOCKING (esp. bleater-env unreachable) → fix → PHASE 0
  ▼
PHASE 3: Rollout analysis ─────────────────────────────────────────
  pull ≥5 rollouts; read 2 transcripts (1 pass,1 fail); run script
  → avg < <TARGET_MEAN> AND both vary AND not perfectly correlated → DONE
  → else → maddy-score-tuner → fix → PHASE 0
  ▼
DONE
```

---

## Phase 0 — Pre-push commands

```bash
cd /Users/mac/Documents/tasks

echo "=== Dockerfile ===" && cat tasks/maddy-delivery-split-brain/Dockerfile
#   Must NOT contain: ENABLE_ISTIO_BLEATER
#   Must contain:     ALLOWED_NAMESPACES (incl. kube-system; verify bleater-env reachability on live container)
#   Must contain:     COPY data/ubuntu-user-rbac.yaml

bash -n tasks/maddy-delivery-split-brain/setup.sh    && echo "setup.sh OK"
bash -n tasks/maddy-delivery-split-brain/solution.sh && echo "solution.sh OK"
python3 -m py_compile tasks/maddy-delivery-split-brain/grader.py && echo "grader.py OK"

source horizon_env/bin/activate
horizon check-anatomy tasks/maddy-delivery-split-brain 2>&1
horizon check-quality tasks/maddy-delivery-split-brain 2>&1 | tail -30
```

## Phase 1 — Push

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks push tasks/maddy-delivery-split-brain 2>&1
# Record: "✓ New version pushed successfully! Version: NNN"
```

## Phase 2 — Oracle validation

**Do NOT rely on `--wait` alone** (TTY spinners buffer and the foreground wait can hang after the hosted run finished). Poll via `validate-logs`:

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate

horizon tasks validate -m hosted -a oracle tasks/maddy-delivery-split-brain 2>&1
# Capture Build ID: "val-1c67bc57-<timestamp>" (or derive from the .validation/ subdir name)
BUILD_ID="val-1c67bc57-<TIMESTAMP_FROM_OUTPUT>"
RESULT_PATH="tasks/maddy-delivery-split-brain/.validation/${BUILD_ID}/result.json"

for attempt in $(seq 1 30); do
  sleep 60
  horizon tasks validate-logs -a oracle tasks/maddy-delivery-split-brain >/dev/null 2>&1
  STATUS=$(python3 -c "import json; print(json.load(open('$RESULT_PATH')).get('status','unknown'))" 2>/dev/null)
  if [ "$STATUS" != "running" ] && [ -n "$STATUS" ]; then
    echo "Oracle completed after ${attempt} min, status=$STATUS"
    cat "$RESULT_PATH" | python3 -m json.tool
    break
  fi
  echo "[poll ${attempt}/30] status=$STATUS"
done
```

**Pass:** `score: 1.0`, `passed: true`, both `delivery_consistency=1` and `observability_governance=1`. Run via a background bash task (`run_in_background: true`) and let the harness notify — don't foreground-wait in a 2-minute Bash call.

Anything else → invoke `maddy-validation-debugger` (do NOT edit files directly first).

## Phase 2.5 — Live red-team

Invoke `horizon-agentic-reviewer` against UUID `<TASK_ID>`. The reviewer spins up a live container, runs `horizon setup`, **executes setup.sh manually** (horizon setup does NOT — see horizon-agentic-reviewer Step 3.5), probes as `ubuntu`, runs the 24-point checklist.

Beyond the standard checklist, ensure these maddy-local hypotheses (exact probes in `maddy-task-guardian` Step 4b):
- **bleater-env is reachable by `ubuntu`** (the #1 structural risk — if Forbidden, governance Layer 6 is unsolvable)
- All three drift controllers exist and are annotation/sidecar discoverable (kube-system, bleater-env, canary)
- Drift names are NOT leaked in the baseline config
- `maddy-gitops-source` is stale (`replicas: 1`, `mode: direct`)
- A no-op agent passes neither subscore

**Phase 2.5 is auto-invoked.** Treat any BLOCKING finding as a failure → auto-fix → re-loop. **Skip Phase 2.5 only** for a pure numeric tweak inside existing structure (e.g. a drift `sleep` value or a `wait_consistent` shift within ceiling). Any change that adds/removes resources, touches RBAC, or changes `ALLOWED_NAMESPACES` requires Phase 2.5.

If SSH to the VM fails, record the failure in the handoff message and proceed to Phase 3 with a noted risk — don't block the loop on environment issues.

**Tell the user (verbatim, the ONLY user-facing message in the loop):**
> "Oracle passed at version N and live red-team is [clean | skipped: reason]. Please trigger a rollout batch for version N in the Horizon UI."

## Phase 3 — Rollout analysis

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon rollouts pull --version NNN tasks/maddy-delivery-split-brain 2>&1
```

Poll every few minutes until ≥5 rollouts download. **Read 2 transcripts (1 pass, 1 fail) before running the script** — numbers tell you the difficulty; transcripts tell you why.

### Analysis script

```python
import json, glob
from collections import defaultdict

VERSION = "NNN"  # replace
TASK = "maddy-delivery-split-brain"
files = glob.glob(f"tasks/{TASK}/.rollouts/v{VERSION}/*.json")

scores = []
sub_vals = defaultdict(list)
joint = defaultdict(int)   # correlation tracker: (consistency, governance) -> count

for f in files:
    d = json.load(open(f))
    scores.append(d["score"])
    try:
        gr = json.loads(d.get("grade_result", "{}"))
        subs = gr.get("subscores", {})
        for k, v in subs.items():
            sub_vals[k].append(v)
        c = subs.get("delivery_consistency"); g = subs.get("observability_governance")
        if c is not None and g is not None:
            joint[(c, g)] += 1
    except Exception:
        pass

if not scores:
    print("No rollouts found — trigger an eval batch first")
else:
    n = len(scores); avg = sum(scores)/n
    pass_rate = sum(1 for s in scores if s >= 0.99)/n
    print(f"N={n}  avg={avg:.3f}  pass_rate={pass_rate:.1%}")
    for k, vs in sub_vals.items():
        vals = sorted(set(vs)); status = "DEAD" if len(vals)==1 else "varies"
        print(f"  {k}: {status} {vals}  mean={sum(vs)/len(vs):.2f}")
    print("  joint (consistency,governance):", dict(joint))
    half = sum(1 for s in scores if abs(s-0.5) < 0.01)
    correlated = (half == 0)  # no 0.5 rollouts => subscores move together
    if avg >= <TARGET_MEAN>:
        print("HIGH: avg >= <TARGET_MEAN> — invoke maddy-score-tuner")
    elif any(len(set(v))==1 for v in sub_vals.values()):
        print("DEADWEIGHT: a subscore is stuck — invoke maddy-score-tuner")
    elif correlated:
        print("CORRELATION COLLAPSE: no 0.5 rollouts — subscores move together — invoke maddy-score-tuner (Coupling/Decorrelate)")
    else:
        print("DONE: avg < <TARGET_MEAN>, both subscores vary, decorrelated")
```

### Decision rules (Phase 3 outcomes)

| Rollout result | Action |
|---|---|
| avg < <TARGET_MEAN>, both vary, some 0.5 rollouts | **DONE** |
| avg < <TARGET_MEAN> but a subscore deadweight | `maddy-score-tuner` — identify which is too easy/impossible |
| No 0.5 rollouts (perfect correlation) | `maddy-score-tuner` — add a decorrelating lever (don't pull a shared-drift lever) |
| `observability_governance` always 0.0 | **Suspect bleater-env reachability first** (Dockerfile), then `maddy-score-tuner` |
| `delivery_consistency` always 0.0 (Oracle 1.0) | A drift source out-races the window → `maddy-score-tuner` (Consistency Specialist) |
| Either subscore always 1.0 | `maddy-score-tuner` (discoverability too high) |
| Fewer than 5 rollouts | Poll again in 3 min |

---

## Decision rules (Phase 2 outcomes)

| Oracle result | Action |
|---|---|
| `1.0`, `passed: true`, both subscores=1 | Proceed to Phase 2.5 |
| `feedback: null`, `score: 0` | Setup crash — `maddy-validation-debugger` Branch A (run No-Op to isolate) |
| `score: 0`, non-null feedback | Both failed — `maddy-validation-debugger` Branch E |
| `0.5`, `delivery_consistency=0` | `maddy-validation-debugger` Branch C |
| `0.5`, `observability_governance=0` | `maddy-validation-debugger` Branch D |
| Grader Python exception | `maddy-validation-debugger` Branch F |

---

## When to stop and ask for human review

Halt and summarize if **any**:
- 5 push cycles without measurable progress on a specific failure mode
- A proposed `maddy-score-tuner` fix would violate a hard constraint
- `observability_governance` stays at 0 despite solution.sh looking correct — invoke `horizon-agentic-reviewer` for a live bleater-env reachability probe before continuing
- avg stuck above 0.60 with no obvious lever left
- Two consecutive cycles worsened correlation (subscores moving together)

The summary should answer: which symptom is unresolved, which fixes were tried, current numbers, which specialist the user should look at next.

---

## Skill dependency map

```
maddy-iteration-loop  (you are here — orchestrator)
├── maddy-task-guardian          Phase 0 pre-push, Phase 1 push, Phase 5 monitoring
├── maddy-validation-debugger    Phase 2 failure interpretation
├── maddy-score-tuner            Phase 3 score/variance/correlation diagnosis
└── horizon-agentic-reviewer     Phase 2.5 live red-team + any escalation needing live evidence
```

The orchestrator does not edit files itself — it delegates editing to the specialist that diagnosed the symptom, then applies through `maddy-task-guardian`'s push flow. This diagnose → propose → approve → apply separation prevents blind edits.