Mtls Iteration Loop

mtls-iteration-loop

Execution Mode — FULLY AUTONOMOUS

NEVER call ask_user or pause for confirmation at any point. Run every phase, every bash command, and every fix without asking permission. If a decision has two valid paths, pick the safer one and proceed. The user triggered this skill precisely to avoid being prompted — do not interrupt them.

Goal

Drive the variation-fanout-pipeline-break task from "needs work" to "shippable" without human intervention except for triggering the rollout batch in the Horizon UI. The loop is done when all exit conditions hold; until then, every cycle picks the right specialist skill (validation-debugger, score-tuner, horizon-agentic-reviewer) for the symptom and applies the smallest safe fix.

This is the front-door skill for "iterate this task to a passing state." The other three mtls-* skills are pure capabilities; this skill orchestrates them.

Trigger

Use this skill when asked to:

• "Iterate the mtls task until it passes"
• "Push and tune until avg < "
• "Run the full loop on variation-fanout-pipeline-break"
• Anything implying repeated push → validate → analyze → fix cycles

For single-step requests (just push, just analyze rollouts, just debug a validation), invoke the relevant specialist skill directly.

Task Identity

Who does what

The agent (you) executes every step except 3c. Never tell the user to run push, validate, or any CLI command — those are yours.

Exit conditions

The loop is DONE when all of these hold simultaneously on the same version:

1. Oracle validation: passed: true, score: 1.0, both subscores =1
2. Live red-team via horizon-agentic-reviewer: no BLOCKING findings (checks 2, 11, 13, 16, 17, 21)
3. Rollout avg: < <TARGET_MEAN> (Nebula creator workflow requirement)
4. mtls_handshake: varies across the rollout batch — both 0 and 1 appear
5. trust_governance: varies across the rollout batch — both 0 and 1 appear
6. Local quality check: 18/20 (or 15+/20 with only the four known noise checks failing — see mtls-task-guardian Step 2)

Anything short of all six → loop continues.

Loop limit

Do not exceed 5 push cycles without human review. After 5 cycles with no measurable progress on a specific failure mode, stop and write a summary of what was tried and what is still failing. The user is faster than 5 more cycles at that point.

No-Op default

Do NOT run No-Op validation as part of the loop. Oracle only. Oracle exercises the same setup.sh path No-Op does (Oracle = setup.sh + solution.sh + grader). If Oracle scores 1.0 cleanly with both subscores recovered, setup.sh ran fine — No-Op adds no signal.

Run No-Op manually only when Oracle returns feedback: null or score: 0 with no grader detail — that pattern indicates a setup crash, and No-Op isolates whether it's setup or solution.sh interference.

The loop

START
  │
  ▼
┌─ PHASE 0: Pre-push checks ──────────────────────────────────────┐
│  • Dockerfile invariants (mtls-task-guardian Step 0)            │
│  • bash -n setup.sh && bash -n solution.sh && py_compile grader │
│  • horizon check-anatomy (must pass clean)                      │
│  • horizon check-quality (expect 18/20; accept 15+/20 if only   │
│    the four known noise checks fail)                            │
│  If anything else fails → fix → re-run PHASE 0                  │
└─────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─ PHASE 1: Push ─────────────────────────────────────────────────┐
│  horizon tasks push tasks/variation-fanout-pipeline-break   │
│  → record version number NNN                                    │
└─────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─ PHASE 2: Oracle validation ────────────────────────────────────┐
│  horizon tasks validate -m hosted -a oracle --wait              │
│  → Oracle 1.0 + both subscores=1 → PHASE 2.5                    │
│  → anything else → invoke mtls-validation-debugger → fix →      │
│    back to PHASE 0                                              │
└─────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─ PHASE 2.5: Live red-team (mtls-task-guardian Step 4b) ─────────┐
│  Why: 10 min here saves a 60 min rollout cycle.                 │
│  When: every push where setup.sh, grader.py, or task.yaml       │
│        changed. Skip only for pure grader-timing tuning with    │
│        no structural change.                                    │
│  How: invoke horizon-agentic-reviewer on task UUID              │
│       <TASK_ID>.                     │
│  → no BLOCKING findings → ask user to trigger rollout batch     │
│  → BLOCKING finding → fix → back to PHASE 0                     │
└─────────────────────────────────────────────────────────────────┘
  │
  ▼
┌─ PHASE 3: Rollout analysis ─────────────────────────────────────┐
│  horizon rollouts pull --version NNN  (re-run until ≥5 rollouts)│
│  Read at least 2 transcripts (one pass, one fail) BEFORE tuning │
│  Run analysis script (below)                                    │
│  → avg < <TARGET_MEAN> AND both subscores vary → DONE                    │
│  → anything else → invoke mtls-score-tuner → fix → PHASE 0      │
└─────────────────────────────────────────────────────────────────┘
  │
  ▼
DONE

Phase 0 — Pre-push commands

cd /Users/mac/Documents/tasks

# 1. Dockerfile invariants (see mtls-task-guardian Step 0)
echo "=== Dockerfile ===" && cat tasks/variation-fanout-pipeline-break/Dockerfile
#   Must NOT contain: ENABLE_ISTIO_BLEATER
#   Must contain:     ALLOWED_NAMESPACES="kube-system"
#   Must contain:     COPY data/ubuntu-user-rbac.yaml

# 2. Syntax
bash -n tasks/variation-fanout-pipeline-break/setup.sh    && echo "setup.sh OK"
bash -n tasks/variation-fanout-pipeline-break/solution.sh && echo "solution.sh OK"
python3 -m py_compile tasks/variation-fanout-pipeline-break/grader.py && echo "grader.py OK"

# 3. Anatomy + quality
source horizon_env/bin/activate
horizon check-anatomy tasks/variation-fanout-pipeline-break 2>&1
horizon check-quality tasks/variation-fanout-pipeline-break 2>&1 | tail -25

Phase 1 — Push

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks push tasks/variation-fanout-pipeline-break 2>&1
# Record: "✓ New version pushed successfully! Version: NNN"

Phase 2 — Oracle validation

Do NOT rely on --wait alone. The horizon tasks validate --wait CLI uses TTY progress spinners that don't flush properly when running non-interactively in background. The completion message gets buffered and never written to the output file until the parent exits, so the foreground "wait" can hang for 30+ minutes after the hosted run has actually completed. Use a poll-via-validate-logs pattern instead.

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate

# 1. Trigger validation (no --wait, returns immediately)
horizon tasks validate -m hosted -a oracle tasks/variation-fanout-pipeline-break 2>&1
# Capture the Build ID from output: "Build ID: val-6a7cfb6a-<timestamp>"
# (you can also derive it later from the .validation/ subdirectory name)
BUILD_ID="val-6a7cfb6a-<TIMESTAMP_FROM_OUTPUT>"
RESULT_PATH="tasks/variation-fanout-pipeline-break/.validation/${BUILD_ID}/result.json"

# 2. Poll every 60s by running validate-logs.  CRITICAL: the result.json is
#    created early in the build lifecycle with `status: "running"` and updated
#    later when the run completes.  Do NOT exit the poll on file existence
#    alone — check that status != "running" (i.e. "passed" / "failed").
for attempt in $(seq 1 30); do
  sleep 60
  horizon tasks validate-logs -a oracle tasks/variation-fanout-pipeline-break >/dev/null 2>&1
  STATUS=$(python3 -c "import json; print(json.load(open('$RESULT_PATH')).get('status', 'unknown'))" 2>/dev/null)
  if [ "$STATUS" != "running" ] && [ -n "$STATUS" ]; then
    echo "Oracle completed after ${attempt} minute(s), status=$STATUS"
    cat "$RESULT_PATH" | python3 -m json.tool
    break
  fi
  echo "[poll ${attempt}/30] status=$STATUS"
done

Pass criterion: score: 1.0, passed: true, both mtls_handshake=1 and trust_governance=1.

Operational note: when running this via a background bash task, use run_in_background: true and let the harness notify on completion — do not foreground-wait inside a 2-minute Bash tool call.

Anything else → invoke mtls-validation-debugger (do NOT edit files directly first).

Phase 2.5 — Live red-team

Invoke horizon-agentic-reviewer against task UUID <TASK_ID>. The reviewer will:

1. Spin up a live container on the Nebula Aurora VM (ssh nebula-vm)
2. Run horizon setup against the latest version
3. Execute the task's setup.sh manually (horizon setup does NOT do this automatically — see horizon-agentic-reviewer.md Step 3.5)
4. Probe the container as the agent user (ubuntu for this task — it's on the nebula-devops image lineage)
5. Run the 24-point checklist

In addition to the standard checklist, ensure these task-local hypotheses are checked (see mtls-task-guardian Step 4b for the exact probes):

• All four drift sources are discoverable to an ubuntu-perspective audit
• Strategy A (bleater-profile-cache-sync) is hidden — not named in baseline config
• HPA pinner on bleater-cert-reaper is in place
• A no-op agent cannot pass either subscore

Phase 2.5 is auto-invoked. Do not ask the user for approval — just run it. The reviewer writes findings.json; read it programmatically. Treat any BLOCKING finding (checks 2, 11, 13, 16, 17, 21 — see horizon-agentic-reviewer's coverage map) as a failure → auto-fix and re-loop. Non-blocking findings can be deferred but should be noted in the rollout-handoff message.

Skip Phase 2.5 only when the change is purely a numeric tweak inside an existing structure (e.g. changing time.sleep(45) to time.sleep(30), or wait_consistent parameter shifts within ceiling). Any change that adds/removes resources, modifies metadata (finalizers, labels, owner references), or touches RBAC requires Phase 2.5.

If SSH to the Nebula Aurora VM fails (auth / connectivity), record the failure mode in the rollout-handoff message and proceed to Phase 3 with a noted risk — do NOT block the loop on environment issues. The rollout itself is the next-best-evidence ground truth.

Only after Phase 2.5 is clean (or skipped per the rules above) do you proceed to Phase 3.

Tell the user (verbatim, the ONLY user-facing message in the loop):

"Oracle passed at version N and live red-team is [clean | skipped: reason]. Please trigger a rollout batch for version N in the Horizon UI."

Phase 3 — Rollout analysis

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate

# Replace NNN with the version pushed in Phase 1
horizon rollouts pull --version NNN \
  tasks/variation-fanout-pipeline-break 2>&1

Poll every few minutes until at least 5 rollouts have downloaded. Then read 2 transcripts before running the script — one pass, one fail. The numbers tell you whether the task is at the right difficulty; the transcripts tell you why agents are succeeding or failing.

Analysis script

Save as /tmp/analyze_rollouts.py or paste into a Python REPL. Replace NNN with the current version.

import json, glob
from collections import defaultdict

VERSION = "NNN"  # replace
TASK = "variation-fanout-pipeline-break"
files = glob.glob(f"tasks/{TASK}/.rollouts/v{VERSION}/*.json")

scores = []
sub_vals = defaultdict(list)

for f in files:
    d = json.load(open(f))
    scores.append(d["score"])
    try:
        gr = json.loads(d.get("grade_result", "{}"))
        for k, v in gr.get("subscores", {}).items():
            sub_vals[k].append(v)
    except Exception:
        pass

if not scores:
    print("No rollouts found — trigger an eval batch first")
else:
    n = len(scores)
    avg = sum(scores) / n
    pass_rate = sum(1 for s in scores if s >= 0.99) / n
    print(f"N={n}  avg={avg:.3f}  pass_rate={pass_rate:.1%}")
    print()
    for k, vs in sub_vals.items():
        vals = sorted(set(vs))
        status = "DEAD" if len(vals) == 1 else "varies"
        mean = sum(vs) / len(vs)
        print(f"  {k}: {status} {vals}  mean={mean:.2f}")
    print()
    if avg >= <TARGET_MEAN>:
        print("HIGH: avg >= <TARGET_MEAN> — invoke mtls-score-tuner")
    elif any(len(set(v)) == 1 for v in sub_vals.values()):
        print("DEADWEIGHT: a subscore is stuck — invoke mtls-score-tuner")
    else:
        print("DONE: avg < <TARGET_MEAN> and both subscores vary")

Decision rules (Phase 3 outcomes)

Decision rules (Phase 2 outcomes)

When to stop and ask for human review

Halt the loop and write a summary if any of these:

• 5 push cycles completed without measurable progress on a specific failure mode
• A proposed fix from mtls-score-tuner would violate a hard constraint (see mtls-task-guardian Hard constraints table)
• Oracle consistently returns score: 0.5 on trust_governance despite solution.sh appearing correct — invoke horizon-agentic-reviewer for a live probe before continuing
• Rollout avg is stuck above 0.60 despite no obvious tuning lever remaining
• Two consecutive cycles introduced coupling (changing handshake moved governance, or vice versa)

The summary should answer: which symptom is unresolved, which fixes were tried, what the current numbers are, and which specialist role the user should look at next.

Skill dependency map

mtls-iteration-loop  (you are here — orchestrator)
├── mtls-task-guardian          Phase 0 pre-push checks, Phase 1 push, Phase 4 monitoring
├── mtls-validation-debugger    Phase 2 failure interpretation
├── mtls-score-tuner            Phase 3 score/variance diagnosis and lever selection
└── horizon-agentic-reviewer    Phase 2.5 live red-team and any escalation requiring live evidence

The orchestrator does not edit files itself — it delegates editing to the specialist that diagnosed the symptom. The specialist proposes the edit; the orchestrator (this skill) applies it through mtls-task-guardian's push flow. This separation prevents the "blind edit" failure mode that wasted cycles in earlier versions.