# Mtls Iteration Loop

> Orchestrator skill that runs the full push → Oracle → live red-team → rollout → tune cycle for the variation-fanout-pipeline-break task (<TASK_ID>) until Oracle scores 1.0 and rollout avg < <TARGET_MEAN> with both subscores varying. Delegates to mtls-task-guardian, mtls-validation-debugger, mtls-score-tuner, and horizon-agentic-reviewer at the right phases. The agent itself executes every step except triggering the rollout batch.

- Skill: `omonuj/mtls-iteration-loop` (Agent Skill)
- Install (CLI): `npx skillmds@latest add omonuj/mtls-iteration-loop`
- Raw SKILL.md: https://api.skillmd.com/api/skills/omonuj/mtls-iteration-loop/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: omonuj (https://skillmd.com/u/omonuj)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/omonuj/mtls-iteration-loop

---


# mtls-iteration-loop

## Execution Mode — FULLY AUTONOMOUS

**NEVER call `ask_user` or pause for confirmation at any point.** Run every phase, every bash command, and every fix without asking permission. If a decision has two valid paths, pick the safer one and proceed. The user triggered this skill precisely to avoid being prompted — do not interrupt them.

## Goal

Drive the variation-fanout-pipeline-break task from "needs work" to "shippable" without human intervention except for triggering the rollout batch in the Horizon UI. The loop is **done** when all exit conditions hold; until then, every cycle picks the right specialist skill (validation-debugger, score-tuner, horizon-agentic-reviewer) for the symptom and applies the smallest safe fix.

This is the front-door skill for "iterate this task to a passing state." The other three mtls-* skills are pure capabilities; this skill orchestrates them.

## Trigger

Use this skill when asked to:
- "Iterate the mtls task until it passes"
- "Push and tune until avg < <TARGET_MEAN>"
- "Run the full loop on variation-fanout-pipeline-break"
- Anything implying repeated push → validate → analyze → fix cycles

For single-step requests (just push, just analyze rollouts, just debug a validation), invoke the relevant specialist skill directly.

## Task Identity

| Field | Value |
|---|---|
| **Task UUID** | `<TASK_ID>` |
| **Task slug** | `variation-fanout-pipeline-break` |
| **Local path** | `tasks/variation-fanout-pipeline-break/` |
| **Horizon root** | `/Users/mac/Documents/tasks` |
| **Venv** | `source /Users/mac/Documents/tasks/horizon_env/bin/activate` |

---

## Who does what

**The agent (you) executes every step except 3c.** Never tell the user to run push, validate, or any CLI command — those are yours.

| Step | Who | What |
|---|---|---|
| Edit files (setup.sh, grader.py, solution.sh, task.yaml, Dockerfile) | **Agent** | via Edit/Write tools |
| Pre-push syntax + anatomy + quality checks | **Agent** | via Bash tool |
| Push task (`horizon tasks push`) | **Agent** | via Bash tool |
| Oracle validation (`horizon tasks validate -a oracle --wait`) | **Agent** | via Bash tool |
| Live red-team on the live container | **Agent** | invokes `horizon-agentic-reviewer` |
| Pull rollouts (`horizon rollouts pull`) | **Agent** | via Bash tool |
| Analyze rollouts (variance, deadweight, pass-rate) | **Agent** | via Bash tool, script below |
| Diagnose & propose tuning fix | **Agent** | invokes `mtls-score-tuner` |
| Diagnose & propose validation fix | **Agent** | invokes `mtls-validation-debugger` |
| **Trigger the eval batch in the Horizon UI** | **User** | the only manual step |

---

## Exit conditions

The loop is DONE when **all** of these hold simultaneously on the same version:

1. Oracle validation: `passed: true, score: 1.0`, both subscores `=1`
2. Live red-team via `horizon-agentic-reviewer`: no BLOCKING findings (checks 2, 11, 13, 16, 17, 21)
3. Rollout avg: `< <TARGET_MEAN>` (Nebula creator workflow requirement)
4. `mtls_handshake`: varies across the rollout batch — both 0 and 1 appear
5. `trust_governance`: varies across the rollout batch — both 0 and 1 appear
6. Local quality check: `18/20` (or `15+/20` with only the four known noise checks failing — see `mtls-task-guardian` Step 2)

Anything short of all six → loop continues.

## Loop limit

**Do not exceed 5 push cycles without human review.** After 5 cycles with no measurable progress on a specific failure mode, stop and write a summary of what was tried and what is still failing. The user is faster than 5 more cycles at that point.

## No-Op default

**Do NOT run No-Op validation as part of the loop. Oracle only.** Oracle exercises the same setup.sh path No-Op does (Oracle = setup.sh + solution.sh + grader). If Oracle scores 1.0 cleanly with both subscores recovered, setup.sh ran fine — No-Op adds no signal.

Run No-Op manually **only** when Oracle returns `feedback: null` or `score: 0` with no grader detail — that pattern indicates a setup crash, and No-Op isolates whether it's setup or solution.sh interference.

---

## The loop

```
START
  │
  ▼
┌─ PHASE 0: Pre-push checks ──────────────────────────────────────┐
│  • Dockerfile invariants (mtls-task-guardian Step 0)            │
│  • bash -n setup.sh && bash -n solution.sh && py_compile grader │
│  • horizon check-anatomy (must pass clean)                      │
│  • horizon check-quality (expect 18/20; accept 15+/20 if only   │
│    the four known noise checks fail)                            │
│  If anything else fails → fix → re-run PHASE 0                  │
└─────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─ PHASE 1: Push ─────────────────────────────────────────────────┐
│  horizon tasks push tasks/variation-fanout-pipeline-break   │
│  → record version number NNN                                    │
└─────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─ PHASE 2: Oracle validation ────────────────────────────────────┐
│  horizon tasks validate -m hosted -a oracle --wait              │
│  → Oracle 1.0 + both subscores=1 → PHASE 2.5                    │
│  → anything else → invoke mtls-validation-debugger → fix →      │
│    back to PHASE 0                                              │
└─────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─ PHASE 2.5: Live red-team (mtls-task-guardian Step 4b) ─────────┐
│  Why: 10 min here saves a 60 min rollout cycle.                 │
│  When: every push where setup.sh, grader.py, or task.yaml       │
│        changed. Skip only for pure grader-timing tuning with    │
│        no structural change.                                    │
│  How: invoke horizon-agentic-reviewer on task UUID              │
│       <TASK_ID>.                     │
│  → no BLOCKING findings → ask user to trigger rollout batch     │
│  → BLOCKING finding → fix → back to PHASE 0                     │
└─────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─ PHASE 3: Rollout analysis ─────────────────────────────────────┐
│  horizon rollouts pull --version NNN  (re-run until ≥5 rollouts)│
│  Read at least 2 transcripts (one pass, one fail) BEFORE tuning │
│  Run analysis script (below)                                    │
│  → avg < <TARGET_MEAN> AND both subscores vary → DONE                    │
│  → anything else → invoke mtls-score-tuner → fix → PHASE 0      │
└─────────────────────────────────────────────────────────────────┘
  │
  ▼
DONE
```

---

## Phase 0 — Pre-push commands

```bash
cd /Users/mac/Documents/tasks

# 1. Dockerfile invariants (see mtls-task-guardian Step 0)
echo "=== Dockerfile ===" && cat tasks/variation-fanout-pipeline-break/Dockerfile
#   Must NOT contain: ENABLE_ISTIO_BLEATER
#   Must contain:     ALLOWED_NAMESPACES="kube-system"
#   Must contain:     COPY data/ubuntu-user-rbac.yaml

# 2. Syntax
bash -n tasks/variation-fanout-pipeline-break/setup.sh    && echo "setup.sh OK"
bash -n tasks/variation-fanout-pipeline-break/solution.sh && echo "solution.sh OK"
python3 -m py_compile tasks/variation-fanout-pipeline-break/grader.py && echo "grader.py OK"

# 3. Anatomy + quality
source horizon_env/bin/activate
horizon check-anatomy tasks/variation-fanout-pipeline-break 2>&1
horizon check-quality tasks/variation-fanout-pipeline-break 2>&1 | tail -25
```

## Phase 1 — Push

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks push tasks/variation-fanout-pipeline-break 2>&1
# Record: "✓ New version pushed successfully! Version: NNN"
```

## Phase 2 — Oracle validation

**Do NOT rely on `--wait` alone.** The `horizon tasks validate --wait` CLI uses TTY progress spinners that don't flush properly when running non-interactively in background. The completion message gets buffered and never written to the output file until the parent exits, so the foreground "wait" can hang for 30+ minutes after the hosted run has actually completed. Use a poll-via-`validate-logs` pattern instead.

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate

# 1. Trigger validation (no --wait, returns immediately)
horizon tasks validate -m hosted -a oracle tasks/variation-fanout-pipeline-break 2>&1
# Capture the Build ID from output: "Build ID: val-6a7cfb6a-<timestamp>"
# (you can also derive it later from the .validation/ subdirectory name)
BUILD_ID="val-6a7cfb6a-<TIMESTAMP_FROM_OUTPUT>"
RESULT_PATH="tasks/variation-fanout-pipeline-break/.validation/${BUILD_ID}/result.json"

# 2. Poll every 60s by running validate-logs.  CRITICAL: the result.json is
#    created early in the build lifecycle with `status: "running"` and updated
#    later when the run completes.  Do NOT exit the poll on file existence
#    alone — check that status != "running" (i.e. "passed" / "failed").
for attempt in $(seq 1 30); do
  sleep 60
  horizon tasks validate-logs -a oracle tasks/variation-fanout-pipeline-break >/dev/null 2>&1
  STATUS=$(python3 -c "import json; print(json.load(open('$RESULT_PATH')).get('status', 'unknown'))" 2>/dev/null)
  if [ "$STATUS" != "running" ] && [ -n "$STATUS" ]; then
    echo "Oracle completed after ${attempt} minute(s), status=$STATUS"
    cat "$RESULT_PATH" | python3 -m json.tool
    break
  fi
  echo "[poll ${attempt}/30] status=$STATUS"
done
```

**Pass criterion:** `score: 1.0`, `passed: true`, both `mtls_handshake=1` and `trust_governance=1`.

**Operational note:** when running this via a background bash task, use `run_in_background: true` and let the harness notify on completion — do not foreground-wait inside a 2-minute Bash tool call.

Anything else → invoke `mtls-validation-debugger` (do NOT edit files directly first).

## Phase 2.5 — Live red-team

Invoke `horizon-agentic-reviewer` against task UUID `<TASK_ID>`. The reviewer will:

1. Spin up a live container on the Nebula Aurora VM (`ssh nebula-vm`)
2. Run `horizon setup` against the latest version
3. **Execute the task's setup.sh manually** (horizon setup does NOT do this automatically — see horizon-agentic-reviewer.md Step 3.5)
4. Probe the container as the agent user (`ubuntu` for this task — it's on the nebula-devops image lineage)
5. Run the 24-point checklist

In addition to the standard checklist, ensure these task-local hypotheses are checked (see `mtls-task-guardian` Step 4b for the exact probes):

- All four drift sources are discoverable to an `ubuntu`-perspective audit
- Strategy A (`bleater-profile-cache-sync`) is hidden — not named in baseline config
- HPA pinner on `bleater-cert-reaper` is in place
- A no-op agent cannot pass either subscore

**Phase 2.5 is auto-invoked.** Do not ask the user for approval — just run it. The reviewer writes `findings.json`; read it programmatically. Treat any BLOCKING finding (checks 2, 11, 13, 16, 17, 21 — see horizon-agentic-reviewer's coverage map) as a failure → auto-fix and re-loop. Non-blocking findings can be deferred but should be noted in the rollout-handoff message.

**Skip Phase 2.5 only when** the change is purely a numeric tweak inside an existing structure (e.g. changing `time.sleep(45)` to `time.sleep(30)`, or `wait_consistent` parameter shifts within ceiling). Any change that adds/removes resources, modifies metadata (finalizers, labels, owner references), or touches RBAC requires Phase 2.5.

If SSH to the Nebula Aurora VM fails (auth / connectivity), record the failure mode in the rollout-handoff message and proceed to Phase 3 with a noted risk — do NOT block the loop on environment issues. The rollout itself is the next-best-evidence ground truth.

Only after Phase 2.5 is clean (or skipped per the rules above) do you proceed to Phase 3.

**Tell the user (verbatim, the ONLY user-facing message in the loop):**
> "Oracle passed at version N and live red-team is [clean | skipped: reason]. Please trigger a rollout batch for version N in the Horizon UI."

## Phase 3 — Rollout analysis

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate

# Replace NNN with the version pushed in Phase 1
horizon rollouts pull --version NNN \
  tasks/variation-fanout-pipeline-break 2>&1
```

Poll every few minutes until at least 5 rollouts have downloaded. Then **read 2 transcripts before running the script** — one pass, one fail. The numbers tell you whether the task is at the right difficulty; the transcripts tell you why agents are succeeding or failing.

### Analysis script

Save as `/tmp/analyze_rollouts.py` or paste into a Python REPL. Replace `NNN` with the current version.

```python
import json, glob
from collections import defaultdict

VERSION = "NNN"  # replace
TASK = "variation-fanout-pipeline-break"
files = glob.glob(f"tasks/{TASK}/.rollouts/v{VERSION}/*.json")

scores = []
sub_vals = defaultdict(list)

for f in files:
    d = json.load(open(f))
    scores.append(d["score"])
    try:
        gr = json.loads(d.get("grade_result", "{}"))
        for k, v in gr.get("subscores", {}).items():
            sub_vals[k].append(v)
    except Exception:
        pass

if not scores:
    print("No rollouts found — trigger an eval batch first")
else:
    n = len(scores)
    avg = sum(scores) / n
    pass_rate = sum(1 for s in scores if s >= 0.99) / n
    print(f"N={n}  avg={avg:.3f}  pass_rate={pass_rate:.1%}")
    print()
    for k, vs in sub_vals.items():
        vals = sorted(set(vs))
        status = "DEAD" if len(vals) == 1 else "varies"
        mean = sum(vs) / len(vs)
        print(f"  {k}: {status} {vals}  mean={mean:.2f}")
    print()
    if avg >= <TARGET_MEAN>:
        print("HIGH: avg >= <TARGET_MEAN> — invoke mtls-score-tuner")
    elif any(len(set(v)) == 1 for v in sub_vals.values()):
        print("DEADWEIGHT: a subscore is stuck — invoke mtls-score-tuner")
    else:
        print("DONE: avg < <TARGET_MEAN> and both subscores vary")
```

### Decision rules (Phase 3 outcomes)

| Rollout result | Action |
|---|---|
| avg < <TARGET_MEAN> AND both subscores vary | **DONE** — exit loop |
| avg < <TARGET_MEAN> BUT a subscore is deadweight | Invoke `mtls-score-tuner` — identify which subscore is too easy/impossible |
| `mtls_handshake` always 1.0 | Invoke `mtls-score-tuner` (Handshake Specialist role) |
| `trust_governance` always 1.0 | Accepted if avg < <TARGET_MEAN>; otherwise invoke `mtls-score-tuner` (Governance Specialist role) |
| `trust_governance` always 0.0 | Likely structural — check `ALLOWED_NAMESPACES` in Dockerfile and kube-system RBAC in setup.sh before invoking `mtls-score-tuner` |
| `mtls_handshake` always 0.0 | Re-verify Oracle first via `mtls-validation-debugger`. If Oracle is 1.0, drift loop is too fast for grader window — invoke `mtls-score-tuner` |
| Fewer than 5 rollouts pulled | Poll again in 3 min — minimum 5 needed for meaningful stats |

---

## Decision rules (Phase 2 outcomes)

| Oracle result | Action |
|---|---|
| `score: 1.0`, `passed: true`, both subscores=1 | Proceed to Phase 2.5 |
| `score: 1.0` but `passed: false` | Grader threshold drift — should not happen with current grader; invoke `mtls-validation-debugger` |
| `feedback: null`, `score: 0` | Setup crash — invoke `mtls-validation-debugger` Branch A |
| `score: 0`, non-null feedback | Both subscores failed — invoke `mtls-validation-debugger` Branch E |
| `score: 0.5`, `mtls_handshake=0`, `trust_governance=1` | Invoke `mtls-validation-debugger` Branch C |
| `score: 0.5`, `mtls_handshake=1`, `trust_governance=0` | Invoke `mtls-validation-debugger` Branch D |
| Grader Python exception | Invoke `mtls-validation-debugger` Branch F |

---

## When to stop and ask for human review

Halt the loop and write a summary if **any** of these:

- 5 push cycles completed without measurable progress on a specific failure mode
- A proposed fix from `mtls-score-tuner` would violate a hard constraint (see `mtls-task-guardian` Hard constraints table)
- Oracle consistently returns `score: 0.5` on `trust_governance` despite solution.sh appearing correct — invoke `horizon-agentic-reviewer` for a live probe before continuing
- Rollout avg is stuck above 0.60 despite no obvious tuning lever remaining
- Two consecutive cycles introduced coupling (changing handshake moved governance, or vice versa)

The summary should answer: which symptom is unresolved, which fixes were tried, what the current numbers are, and which specialist role the user should look at next.

---

## Skill dependency map

```
mtls-iteration-loop  (you are here — orchestrator)
├── mtls-task-guardian          Phase 0 pre-push checks, Phase 1 push, Phase 4 monitoring
├── mtls-validation-debugger    Phase 2 failure interpretation
├── mtls-score-tuner            Phase 3 score/variance diagnosis and lever selection
└── horizon-agentic-reviewer    Phase 2.5 live red-team and any escalation requiring live evidence
```

The orchestrator does not edit files itself — it delegates editing to the specialist that diagnosed the symptom. The specialist proposes the edit; the orchestrator (this skill) applies it through `mtls-task-guardian`'s push flow. This separation prevents the "blind edit" failure mode that wasted cycles in earlier versions.