Istio Iteration Loop

istio-iteration-loop

Execution Mode — FULLY AUTONOMOUS

NEVER call ask_user or pause for confirmation at any point. Run every phase, command, and fix without asking. If a decision has two valid paths, pick the safer one. The user triggered this skill to avoid being prompted — do not interrupt them.

Goal

Drive variation-istio-split-brain from "needs work" to "shippable" with no human step except triggering the rollout batch. Done when all exit conditions hold; until then each cycle picks the right specialist for the symptom and applies the smallest safe fix.

Trigger

"Iterate the istio task until it passes"; "push and tune until avg < "; "run the full loop on variation-istio-split-brain"; anything implying repeated push → validate → analyze → fix. For single steps, invoke the specialist directly.

Task Identity

Who does what

The agent executes everything except triggering the eval batch in the Horizon UI (the only user step). Never tell the user to push/validate/pull — those are yours.

Exit conditions (all, same version)

1. Oracle passed: true, score: 1.0, both subscores =1
2. Live red-team — DISABLED (user directive). The live red-team is nulled for this task; do not run horizon-agentic-reviewer. Go straight from Oracle 1.0 to asking the user to trigger the eval batch.
3. Rollout avg < <TARGET_MEAN> and > <FLOOR> (avg < → over-hard, rubric-concern risk → istio-score-tuner to loosen)
4. routing_restored varies (0 and 1 appear)
5. mesh_governance varies (0 and 1 appear)
6. Decorrelated — some 0.5 rollouts appear (not only 0.0 and 1.0). v2 decorrelates via the two /etc/bleater gates; if 0.5s vanish, those gates regressed (not a cue to invent new ones)
7. Local quality: 18 substantive pass (v1_feedback section may FAIL as accepted noise — incl. complaints about "undisclosed" hidden requirements; do NOT fix by leaking specifics to task.yaml)

Short of all seven → continue.

Loop limit

Max 5 push cycles without human review. Then stop and summarize.

No-Op default

Oracle only. Run No-Op manually only when Oracle returns feedback: null/score: 0 (setup-crash isolation — e.g. a CRD-establishment race).

The loop

PHASE 0  Pre-push checks ───────────────────────────────────
  • Dockerfile + CRD invariants (istio-task-guardian Step 0)
    — incl. install_open_crd + establish-wait intact
  • bash -n setup.sh && bash -n solution.sh && py_compile grader
  • horizon check-anatomy (clean)
  • horizon check-quality (18 substantive pass; v1_feedback may FAIL)
  anything else fails → fix → re-run PHASE 0
PHASE 1  Push → record version NNN
PHASE 2  Oracle validation
  → 1.0 + both=1 → ask user to trigger rollout/eval batch   (Phase 2.5 is DISABLED)
  → else → istio-validation-debugger → fix → PHASE 0
PHASE 2.5 Live red-team — DISABLED (user directive); never run horizon-agentic-reviewer
PHASE 3  Rollout analysis (≥5 rollouts, read 2 transcripts, run script)
  → avg < <TARGET_MEAN> AND both vary AND decorrelated (0.5s exist) → DONE
  → else → istio-score-tuner → fix → PHASE 0

Phase 0 — Pre-push

cd /Users/mac/Documents/tasks
echo "=== Dockerfile ===" && cat tasks/variation-istio-split-brain/Dockerfile
#  must NOT contain ENABLE_ISTIO_BLEATER; must contain ALLOWED_NAMESPACES="kube-system,bleater-env" (BOTH) + COPY data/ubuntu-user-rbac.yaml
grep -nE "install_open_crd|--for=condition=established" tasks/variation-istio-split-brain/setup.sh  # CRD block intact
grep -nE "ubuntu-user-mesh-admission|/etc/bleater|/opt/grader-(install-id|route-policy)" tasks/variation-istio-split-brain/setup.sh  # v2: webhook ClusterRole + artefacts present
bash -n tasks/variation-istio-split-brain/setup.sh    && echo "setup.sh OK"
bash -n tasks/variation-istio-split-brain/solution.sh && echo "solution.sh OK"
python3 -m py_compile tasks/variation-istio-split-brain/grader.py && echo "grader.py OK"
source horizon_env/bin/activate
horizon check-anatomy tasks/variation-istio-split-brain 2>&1
horizon check-quality tasks/variation-istio-split-brain 2>&1 | tail -30

Phase 1 — Push

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks push tasks/variation-istio-split-brain 2>&1   # record Version: NNN

Phase 2 — Oracle (poll, don't trust --wait)

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks validate -m hosted -a oracle tasks/variation-istio-split-brain 2>&1
BUILD_ID="val-f3ce3fde-<TIMESTAMP>"
RESULT_PATH="tasks/variation-istio-split-brain/.validation/${BUILD_ID}/result.json"
for attempt in $(seq 1 30); do
  sleep 60
  horizon tasks validate-logs -a oracle tasks/variation-istio-split-brain >/dev/null 2>&1
  STATUS=$(python3 -c "import json; print(json.load(open('$RESULT_PATH')).get('status','unknown'))" 2>/dev/null)
  if [ "$STATUS" != "running" ] && [ -n "$STATUS" ]; then
    echo "Oracle done after ${attempt} min, status=$STATUS"; cat "$RESULT_PATH" | python3 -m json.tool; break
  fi
  echo "[poll ${attempt}/30] status=$STATUS"
done

Pass: 1.0 + both subscores 1. Run via a background bash task and let the harness notify. Anything else → istio-validation-debugger.

Phase 2.5 — Live red-team — DISABLED (user directive)

Do NOT run horizon-agentic-reviewer. The user runs the equivalent live checks themselves and finds an agent-run pass redundant. The loop goes straight from an Oracle 1.0 (Phase 2) to the user-trigger message below. (The probe list that used to live here is preserved in the horizon-agentic-reviewer SKILL for manual reference only; that skill is nulled and must not be auto-invoked.)

Tell the user (verbatim, the ONLY user-facing message), as soon as Oracle is 1.0:

"Oracle passed at version N. Please trigger a rollout/eval batch for version N in the Horizon UI."

Phase 3 — Rollout analysis

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon rollouts pull --version NNN tasks/variation-istio-split-brain 2>&1

Poll until ≥5 download. Read 2 transcripts (1 pass, 1 fail) before the script.

import json, glob
from collections import defaultdict
VERSION="NNN"; TASK="variation-istio-split-brain"
files=glob.glob(f"tasks/{TASK}/.rollouts/v{VERSION}/*.json")
scores=[]; sub=defaultdict(list); joint=defaultdict(int)
for f in files:
    d=json.load(open(f)); scores.append(d["score"])
    try:
        gr=json.loads(d.get("grade_result","{}")); s=gr.get("subscores",{})
        for k,v in s.items(): sub[k].append(v)
        r=s.get("routing_restored"); g=s.get("mesh_governance")
        if r is not None and g is not None: joint[(r,g)]+=1
    except Exception: pass
if not scores: print("No rollouts — trigger a batch first")
else:
    n=len(scores); avg=sum(scores)/n
    print(f"N={n} avg={avg:.3f} pass_rate={sum(1 for s in scores if s>=0.99)/n:.0%}")
    for k,vs in sub.items():
        vals=sorted(set(vs)); print(f"  {k}: {'DEAD' if len(vals)==1 else 'varies'} {vals} mean={sum(vs)/len(vs):.2f}")
    print("  joint (routing,governance):", dict(joint))
    half=sum(1 for s in scores if abs(s-0.5)<0.01)
    if avg>=<TARGET_MEAN>: print("HIGH avg — istio-score-tuner (tighten ONE /etc/bleater or metadata gate)")
    elif avg < <TARGET_MEAN>: print("LOW avg (<<TARGET_MEAN>) — over-hard, rubric-concern risk — istio-score-tuner (loosen the tightest gate)")
    elif any(len(set(v))==1 for v in sub.values()): print("DEADWEIGHT — istio-score-tuner")
    elif half==0: print("CORRELATION COLLAPSE: no 0.5 rollouts — the /etc/bleater decorrelators regressed; verify they FIRE (setup wrote /etc/bleater + /opt; grader reads /opt) — do NOT invent a new gate")
    else: print("DONE: <FLOOR><avg < <TARGET_MEAN>, both vary, decorrelated")

Decision rules (Phase 2)

When to stop for human review

5 cycles without progress; a tuner fix would violate a hard constraint; mesh_governance stuck at 0 despite a correct-looking solution (investigate manually from the Oracle/rollout logs — live red-team is disabled); avg stuck > with no lever left; two cycles worsened correlation.

Skill dependency map

istio-iteration-loop  (orchestrator)
├── istio-task-guardian          Phase 0 pre-push, Phase 1 push, Phase 5 monitoring
├── istio-validation-debugger    Phase 2 failure interpretation
├── istio-score-tuner            Phase 3 score/variance/correlation diagnosis
└── horizon-agentic-reviewer     DISABLED — live red-team nulled (user directive); do not invoke

The orchestrator delegates editing to the specialist that diagnosed the symptom, then applies via istio-task-guardian. Diagnose → propose → approve → apply.