Istio Task Guardian

istio-task-guardian

Step 0 — QC Spec design gate (READ FIRST, every iteration)

Before editing ANY task file and again before every push, read the QC Spec — .claude/QC-SPEC.md. It is the priority-ordered rubric (Critical → Error → Legitimacy) horizon check-quality and reviewers grade against.

Confirm the intended/current design does not violate any Critical or Error item. The ones this task repeatedly trips:

• No Binary Bundling — never AND heterogeneous checks (routing objects + auth objects + drift + metadata) into one binary subscore. Each subscore = ONE cohesive functional outcome.
• Functional Tests as Subscores — at least one subscore exercises live behavior (the RPC path / JWT validation), not static VirtualService/AuthorizationPolicy field matching.
• Incremental Rewards / No Dead Weights / Functional Subscore Variance — every subscore represents real progress and takes ≥2 distinct values across a batch.
• Equal Weights — subscores equally weighted.
• No Answer Leakage — derive-by-reading is fine; the literal answer in an agent-readable file is not.

If a change would breach a Critical/Error item, stop and rework it before pushing.

Execution Mode — FULLY AUTONOMOUS

NEVER ask the user for confirmation on any terminal command, edit, push, validation, or live-probe. Every bash invocation is pre-approved. The ONLY user-facing handoff is at Step 4c: telling the user to trigger the rollout/eval batch in the Horizon UI (you cannot trigger rollouts from the CLI). Everything else — edit setup.sh / solution.sh / grader.py / task.yaml, run pre-push checks, push, run Oracle validation, poll — runs without asking. (The live red-team is disabled; never invoke horizon-agentic-reviewer.)

If a decision has two valid paths, pick the safer one and proceed.

Goal

Keep variation-istio-split-brain shippable every push: setup.sh runs cleanly (the minimal Istio CRDs install + establish), Oracle scores 1.0, and rollouts average < with both subscores varying AND decorrelated. This is the only skill that pushes the task and the only one allowed to mutate files without first consulting a diagnosis skill — the front door for every other istio-* skill.

Flow note (user directive): the live red-team step is disabled for this task. The flow is: pre-push gates → push → hosted Oracle → (Oracle 1.0) → tell the user to trigger the eval batch. Do NOT invoke horizon-agentic-reviewer as part of the loop. See horizon-agentic-reviewer SKILL — it is nulled.

Trigger

Use when asked to: push and start an eval; run local quality/anatomy/syntax checks; confirm avg rollout < ; walk a pre-push checklist; decide whether to invoke istio-validation-debugger or istio-score-tuner.

Task Identity

Status: built to v2 grader. Hosted Oracle PASSES 1.0 at task version 3 (both subscores 1.0; build val-f3ce3fde-1780317958512). Reaching green took fixing two base-image collisions: (1) real Istio CRDs already ship → install_open_crd must reuse not reinstall; (2) bleater-timeline-service is a pre-existing base-app Service → reuse it, don't recreate (duplicate-port). NOT yet rollout-tuned — the variance numbers below are targets, not measured.

How scoring actually works (read once, internalise)

Always derive constants from grader.py, not from this file. Correct as of grader.py v2.

The scenario is an Istio routing & observability split-brain. No istiod / no Envoy data plane runs (it breaks k3s boot — see the Dockerfile note). setup.sh installs only the minimal Istio CRD schemas (open / preserve-unknown-fields) so the broken mesh objects can be applied and reconciled, and the grader scores config shape, not live traffic.

Two binary subscores, each 0.5:

• routing_restored = 1 (check_routing, wait_consistent(passes=5, gap=8s, timeout=220s)) when ALL hold:
  • the edge EnvoyFilter bleater-edge-timeout is absent OR its route timeout is ≥ 1s (the 0.1s fault is cleared)
  • VirtualService bleater-timeline exists with http routes, and no http route has a sub-second timeout
  • every route destination host resolves to a Service that EXISTS (no bleater-timeline-blackhole... host)
  • no http route delegates to a namespace absent from the cluster (the bleater-routes-ext fault)
  • no http route mirrors to a host with no backing Service (the devnull.blackhole... fault)
  • no ServiceEntry shadows bleater-timeline-service.bleater.svc.cluster.local with STATIC resolution (the DNS-hijack fault)
  • [v4 ROUTING-ONLY DECORRELATOR — multi-step] route-policy subset binding: the timeline route must PIN to a subset (destination.subset), AND a DestinationRule for the timeline host must define that subset with a label whose VALUE equals the contents of /opt/<grader-artefact>.txt (mirror of agent-readable /etc/bleater/<artefact>.txt). Subset name + label key are the agent's choice (loose); the token value + the route→subset→DR→token wiring are strict. This replaced the v2 single-step header stamp (which caught 100% at v3). Documented ONLY in buried baseline-config + the /etc/bleater file. If the artefact file is missing the grader SKIPS this check (no deadweight). Grader fn: <grader-check-fn>.
• mesh_governance = 1 (check_governance, wait_consistent(passes=5, gap=8s, timeout=200s)) when ALL hold:
  • VirtualService carries label bleater.io/<metadata-key>=stable AND annotation bleater.io/<metadata-key>=enforcing (documented only in baseline-config, NOT in task.yaml — thoroughness gate)
  • [v2 GOVERNANCE-ONLY DECORRELATOR] install anchoring: some VirtualService annotation VALUE equals the contents of /opt/<grader-artefact>.txt (mirror of agent-readable /etc/bleater/<artefact>.txt). Annotation KEY is the agent's choice (loose); solution uses bleater.io/<metadata-key>. Documented only in baseline-config + the /etc/bleater file. Skipped if the artefact file is missing.
  • [v2] timeline Service bleater-timeline-service carries annotation bleater.io/<metadata-key>=internal (baseline-documented buried metadata)
  • RequestAuthentication bleater-jwt has a jwtRule with issuer=https://auth.devops.local AND jwksUri=http://auth.devops.local/.well-known/jwks.json, and NO rule whose jwks contains invalid/attacker
  • AuthorizationPolicy bleater-edge-authz is not a blanket DENY (action DENY + empty/match-all rule)
  • the two kube-system drift Deployments, the bleater-env drift Deployment, AND the bleater bleater-mesh-operator Deployment are at replicas=0/absent
  • the mesh-config-refresher CronJob (bleater) is deleted or spec.suspend=true
  • bleater-timeline-canary mutation path neutralized — envoy-patcher sidecar removed OR bleater-timeline-canary-runtime SA/Role/RoleBinding removed (either suffices)
  • [v2] no MutatingWebhookConfiguration still intercepts virtualservices/envoyfilters writes (the admission interceptor must be removed)

score=0 → neither recovered; 0.5 → one; 1.0 → both.

The drift + protection surface (re-applies / re-mutates the broken mesh objects)

Names of the cross-namespace reconcilers are randomized from pools; the grader reads the chosen names from /opt/grader-scenario.json (root-only, keys vs_reconciler, ef_reconciler, env_reconciler). solution.sh rediscovers them by sweeping the known pools.

The VirtualService is re-applied by TWO loop sources (VS reconciler + ENV reconciler) AND re-mutated by the webhook on every write. The EnvoyFilter is re-applied by TWO loop sources (EF reconciler + canary envoy-patcher) AND the webhook. A one-time kubectl apply of a corrected VS/EF is silently mutated back while the webhook exists, so the webhook must be deleted FIRST; then all loop sources stopped; then the objects fixed. solution.sh does exactly this order.

Decorrelation status (read before tuning)

The two subscores were correlated in the v1 design (shared drift surface only). v2 decorrelates them by construction with two hidden /etc/bleater/ artefact gates — <artefact>.txt gives routing_restored independent difficulty; <artefact>.txt gives mesh_governance independent thoroughness difficulty. These plus the VS-metadata gate and the Service annotation are what produce 0.5 rollouts. Do NOT remove or weaken them without a measured reason — they are the variance engine (mirrors the mtls-sibling v28 design that hit avg ). The structural traps (webhook / HPA / finalizer / bleater-env / random names) are the difficulty floor — the strong model defeats them by audit-then-adapt, so they shift avg a little but do not by themselves create variance.

Step 0 — Dockerfile + CRD invariants (ALWAYS check before push)

cat tasks/variation-istio-split-brain/Dockerfile
grep -nE "install_open_crd|--for=condition=established" tasks/variation-istio-split-brain/setup.sh

If any invariant is violated, stop and fix it first. Do not push.

Step 1 — Local syntax + anatomy

cd /Users/mac/Documents/tasks
bash -n tasks/variation-istio-split-brain/setup.sh    && echo "setup.sh OK"
bash -n tasks/variation-istio-split-brain/solution.sh && echo "solution.sh OK"
python3 -m py_compile tasks/variation-istio-split-brain/grader.py && echo "grader.py OK"
# v2: confirm the embedded webhook server compiles (helper image needs python3 to mutate; inert if absent)
awk "/cat > \"\\\$\{WEBHOOK_DIR\}\/webhook.py\" <<'PYEOF'/{f=1;next} /^PYEOF\$/{f=0} f" \
  tasks/variation-istio-split-brain/setup.sh | python3 -m py_compile /dev/stdin && echo "webhook.py OK"
source horizon_env/bin/activate
horizon check-anatomy tasks/variation-istio-split-brain 2>&1

All print OK; anatomy prints Task folder structure is valid!. (The nested .claude/skills/ is a dotfolder — anatomy ignores it, push excludes it.) Any failure stops the workflow.

Step 2 — Local quality check

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon check-quality tasks/variation-istio-split-brain 2>&1 | tail -30

Before push, walk the QC-SPEC.md Critical + Error list against the current grader.py + task.yaml and record pass/fail per item. Every item must pass OR be on the accepted-noise list with a one-line reason. A new failing item that is NOT accepted-noise blocks the push.

Target: 18 substantive checks pass. The v1_feedback extra-strict section may FAIL — accepted noise (the LLM reviewer cannot read the heredoc baseline-config / incident-notes, so it claims they are "missing", dislikes the oracle sweeping the drift pools, and may flag the hidden /etc/bleater requirements as "undisclosed"). Do NOT "fix" any of these by naming drift controllers, the /etc/bleater filenames, the metadata keys, or the route-policy subset/token in task.yaml — that collapses the variance. All requirements ARE disclosed in baseline-config + the agent-readable /etc/bleater/*.txt files; task.yaml legitimately references the conventions + the /etc/bleater/ directory generically.

Step 3 — Push

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks push tasks/variation-istio-split-brain 2>&1

Record Version: NNN. (Server name matches the slug here — no rename caveat, unlike maddy.)

Step 4 — Validation handoff

4a. Oracle (~10–15 min hosted):

horizon tasks validate -m hosted -a oracle --wait tasks/variation-istio-split-brain 2>&1
horizon tasks validate-logs -a oracle tasks/variation-istio-split-brain 2>&1
cat tasks/variation-istio-split-brain/.validation/*/result.json 2>/dev/null | python3 -m json.tool

Pass: score: 1.0, both routing_restored=1 and mesh_governance=1. Anything else → istio-validation-debugger.

v2 Oracle watch-item: the admission webhook depends on python3 being present in ${SIDECAR_IMAGE} (the bleater helper image). If python3 is absent the webhook is inert (failurePolicy: Ignore) but its config object still exists and solution.sh still deletes it → Oracle still 1.0. The Oracle is the first place that confirms whether the webhook actively re-mutates or merely sits inert. Either way the Oracle must hit 1.0; if it does not, read the routing:/governance: feedback halves.

4b. Live red-team — DISABLED (user directive). Do not invoke horizon-agentic-reviewer; the user runs the equivalent checks themselves and finds an agent-run pass redundant. Go straight from an Oracle 1.0 to Step 4c. (The probe list that used to live here is preserved in the horizon-agentic-reviewer SKILL for manual reference only.)

4c. As soon as the Oracle is 1.0 with both subscores =1, tell the user, verbatim:

"Oracle passed at version N. Please trigger a rollout/eval batch for version N in the Horizon UI."

Step 5 — Monitor rollouts

horizon rollouts pull --version NNN tasks/variation-istio-split-brain 2>&1

Re-run until ≥5 download, then use the istio-iteration-loop Phase 3 script.

Target state

Step 6 — Tuning decision

Hard constraints (never violate)

Skill hand-off map

istio-task-guardian (you are here)
├── Step 4a fails  →  istio-validation-debugger
├── Step 4b        →  DISABLED (live red-team nulled — user directive)
├── Step 5 fails   →  istio-score-tuner
└── Step 6 loops   →  istio-iteration-loop (orchestrator)