Oncall Iteration Loop

oncall-iteration-loop

Execution Mode — FULLY AUTONOMOUS

NEVER pause for confirmation at any point. Run every phase, command, and fix without asking. Two valid paths → pick the safer. The only manual step is the user triggering the eval batch in the Horizon UI.

Goal

Drive oncall-schedule-coverage-collapse from "needs work" to "shippable". Done when all exit conditions hold; each cycle picks the right specialist and applies the smallest safe fix.

Trigger

"Iterate the oncall task until it passes"; "push and tune until avg < "; "run the full loop on oncall-schedule-coverage-collapse"; anything implying repeated push → validate → analyze → fix. Single steps → invoke the specialist directly.

Task Identity

Who does what

The agent executes everything except triggering the eval batch in the Horizon UI. Never tell the user to push/validate/pull.

Exit conditions (all, same version)

1. Oracle passed: true, score: 1.0, both subscores =1
2. Live red-team: no BLOCKING findings — especially bleater-env reachable, all 3 drift sources live, install-id agent-readable, a no-op agent failing both
3. Rollout avg < <TARGET_MEAN> (and not < <FLOOR> — over-hard)
4. coverage_restored varies (0 and 1)
5. monitoring_governance varies (0 and 1)
6. Decorrelated — some 0.5 rollouts appear (not only 0.0 and 1.0)
7. Local quality: 18 substantive pass (v1_feedback + behavior_in_task_documentation may FAIL as accepted heredoc-invisibility noise — do NOT fix by leaking the hidden gates to task.yaml)

Short of all seven → continue.

Loop limit

Max 5 push cycles without human review, then stop and summarize.

No-Op default

Oracle only. Run No-Op manually only on feedback: null/score: 0 (setup-crash isolation).

The loop

PHASE 0  Pre-push (oncall-task-guardian Step 0–2): Dockerfile (ALLOWED_NAMESPACES=kube-system,bleater-env)
         + bash -n + py_compile + check-anatomy + check-quality (18 substantive pass)
PHASE 1  Push → version NNN
PHASE 2  Oracle → 1.0+both=1 → PHASE 2.5 ; else → oncall-validation-debugger → fix → PHASE 0
PHASE 2.5 Live red-team (horizon-agentic-reviewer) → clean → ask user to trigger batch ; BLOCKING → fix → PHASE 0
PHASE 3  Rollout analysis → avg < <TARGET_MEAN> AND both vary AND decorrelated (0.5s) → DONE ; else → oncall-score-tuner → PHASE 0

Phase 0 — Pre-push

cd /Users/mac/Documents/tasks
echo "=== Dockerfile ===" && cat tasks/oncall-schedule-coverage-collapse/Dockerfile
#  must NOT contain ENABLE_ISTIO_BLEATER; must contain ALLOWED_NAMESPACES="kube-system,bleater-env" + COPY data/ubuntu-user-rbac.yaml
bash -n tasks/oncall-schedule-coverage-collapse/setup.sh    && echo "setup.sh OK"
bash -n tasks/oncall-schedule-coverage-collapse/solution.sh && echo "solution.sh OK"
python3 -m py_compile tasks/oncall-schedule-coverage-collapse/grader.py && echo "grader.py OK"
source horizon_env/bin/activate
horizon check-anatomy tasks/oncall-schedule-coverage-collapse 2>&1
horizon check-quality tasks/oncall-schedule-coverage-collapse 2>&1 | tail -30

Phase 1 — Push

horizon tasks push tasks/oncall-schedule-coverage-collapse 2>&1   # record Version: NNN

Phase 2 — Oracle (poll, don't trust --wait)

horizon tasks validate -m hosted -a oracle tasks/oncall-schedule-coverage-collapse 2>&1
BUILD_ID="val-9ee1af19-<TIMESTAMP>"
RESULT_PATH="tasks/oncall-schedule-coverage-collapse/.validation/${BUILD_ID}/result.json"
for attempt in $(seq 1 30); do
  sleep 60
  horizon tasks validate-logs -a oracle tasks/oncall-schedule-coverage-collapse >/dev/null 2>&1
  STATUS=$(python3 -c "import json; print(json.load(open('$RESULT_PATH')).get('status','unknown'))" 2>/dev/null)
  if [ "$STATUS" != "running" ] && [ -n "$STATUS" ]; then
    echo "Oracle done after ${attempt} min, status=$STATUS"; cat "$RESULT_PATH" | python3 -m json.tool; break
  fi
  echo "[poll ${attempt}/30] status=$STATUS"
done

Pass: 1.0 + both 1. Run as a background bash task. Else → oncall-validation-debugger.

Phase 2.5 — Live red-team

Invoke horizon-agentic-reviewer on UUID <TASK_ID>. Confirm (probes in oncall-task-guardian Step 4b): bleater-env reachable by ubuntu; the broken state is live; all 3 annotation/slot-pinner drift sources exist (kube-system + bleater-env + canary); /etc/oncall/<artefact>.txt agent-readable but /opt/<grader-artefact>.txt not; drift names + exact metadata keys NOT leaked in baseline; a no-op agent fails both subscores. Auto-invoked. BLOCKING → fix → re-loop. Skip only for a pure numeric tweak. If the VM is unreachable, note the risk and proceed.

Tell the user (verbatim, the ONLY user-facing message):

"Oracle passed at version N and live red-team is [clean | skipped: reason]. Please trigger a rollout batch for version N in the Horizon UI."

Phase 3 — Rollout analysis

horizon rollouts pull --version NNN tasks/oncall-schedule-coverage-collapse 2>&1

Poll until ≥5; read 2 transcripts (1 pass, 1 fail) before the script.

import json, glob
from collections import defaultdict
VERSION="NNN"; TASK="oncall-schedule-coverage-collapse"
files=glob.glob(f"tasks/{TASK}/.rollouts/v{VERSION}/*.json")
scores=[]; sub=defaultdict(list)
for f in files:
    d=json.load(open(f)); scores.append(d["score"])
    try:
        s=json.loads(d.get("grade_result","{}")).get("subscores",{})
        for k,v in s.items(): sub[k].append(v)
    except Exception: pass
if not scores: print("No rollouts — trigger a batch first")
else:
    n=len(scores); avg=sum(scores)/n
    print(f"N={n} avg={avg:.3f} pass_rate={sum(1 for s in scores if s>=0.99)/n:.0%}")
    for k,vs in sub.items():
        vals=sorted(set(vs)); print(f"  {k}: {'DEAD' if len(vals)==1 else 'varies'} {vals} mean={sum(vs)/len(vs):.2f}")
    half=sum(1 for s in scores if abs(s-0.5)<0.01)
    if avg>=<TARGET_MEAN>: print("HIGH avg — oncall-score-tuner")
    elif any(len(set(v))==1 for v in sub.values()): print("DEADWEIGHT — oncall-score-tuner")
    elif half==0: print("CORRELATION COLLAPSE: no 0.5 — oncall-score-tuner (lean on overrides + gap-rule/anchor decorrelators)")
    else: print("DONE: avg < <TARGET_MEAN>, both vary, decorrelated")

Decision rules (Phase 2)

When to stop for human review

5 cycles without progress; a tuner fix would violate a hard constraint; governance stuck at 0 despite a correct-looking solution (run reviewer for a live bleater-env/install-id probe); avg stuck > ; two cycles worsened correlation.

Skill dependency map

oncall-iteration-loop (orchestrator)
├── oncall-task-guardian        Phase 0 pre-push, Phase 1 push, Phase 5 monitoring
├── oncall-validation-debugger  Phase 2 failure interpretation
├── oncall-score-tuner          Phase 3 score/variance/correlation diagnosis
└── horizon-agentic-reviewer    Phase 2.5 live red-team

Diagnose → propose → approve → apply: the orchestrator delegates editing to the specialist that diagnosed the symptom, then applies via oncall-task-guardian.