# Istio Task Guardian

> End-to-end maintenance skill for the variation-istio-split-brain task (<TASK_ID>). Owns the pre-push safety checks, local quality/anatomy gates, push, Oracle validation handoff, and rollout monitoring. Primary goal is to keep avg rollout score < <TARGET_MEAN> with healthy, decorrelated variance across both subscores. Calls istio-validation-debugger when Oracle fails and istio-score-tuner when rollouts regress.

- Skill: `omonuj/istio-task-guardian` (Agent Skill)
- Install (CLI): `npx skillmds@latest add omonuj/istio-task-guardian`
- Raw SKILL.md: https://api.skillmd.com/api/skills/omonuj/istio-task-guardian/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: omonuj (https://skillmd.com/u/omonuj)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/omonuj/istio-task-guardian

---


# istio-task-guardian

## Step 0 — QC Spec design gate (READ FIRST, every iteration)

Before editing ANY task file and again before every push, read the QC Spec — `.claude/QC-SPEC.md`. It is the priority-ordered rubric (Critical → Error → Legitimacy) `horizon check-quality` and reviewers grade against.

Confirm the intended/current design does not violate any Critical or Error item. The ones this task repeatedly trips:
- **No Binary Bundling** — never AND heterogeneous checks (routing objects + auth objects + drift + metadata) into one binary subscore. Each subscore = ONE cohesive functional outcome.
- **Functional Tests as Subscores** — at least one subscore exercises live behavior (the RPC path / JWT validation), not static VirtualService/AuthorizationPolicy field matching.
- **Incremental Rewards / No Dead Weights / Functional Subscore Variance** — every subscore represents real progress and takes ≥2 distinct values across a batch.
- **Equal Weights** — subscores equally weighted.
- **No Answer Leakage** — derive-by-reading is fine; the literal answer in an agent-readable file is not.

If a change would breach a Critical/Error item, stop and rework it before pushing.

## Execution Mode — FULLY AUTONOMOUS

**NEVER ask the user for confirmation on any terminal command, edit, push, validation, or live-probe.** Every bash invocation is pre-approved. The ONLY user-facing handoff is at Step 4c: telling the user to trigger the rollout/eval batch in the Horizon UI (you cannot trigger rollouts from the CLI). Everything else — edit setup.sh / solution.sh / grader.py / task.yaml, run pre-push checks, push, run Oracle validation, poll — runs without asking. (The live red-team is disabled; never invoke `horizon-agentic-reviewer`.)

If a decision has two valid paths, pick the safer one and proceed.

## Goal

Keep variation-istio-split-brain shippable every push: setup.sh runs cleanly (the minimal Istio CRDs install + establish), Oracle scores 1.0, and rollouts average < <TARGET_MEAN> with both subscores varying AND decorrelated. This is the only skill that pushes the task and the only one allowed to mutate files without first consulting a diagnosis skill — the front door for every other istio-* skill.

> **Flow note (user directive):** the live red-team step is **disabled** for this task. The flow is: pre-push gates → push → hosted Oracle → (Oracle 1.0) → **tell the user to trigger the eval batch**. Do NOT invoke `horizon-agentic-reviewer` as part of the loop. See `horizon-agentic-reviewer` SKILL — it is nulled.

## Trigger

Use when asked to: push and start an eval; run local quality/anatomy/syntax checks; confirm avg rollout < <TARGET_MEAN>; walk a pre-push checklist; decide whether to invoke istio-validation-debugger or istio-score-tuner.

## Task Identity

| Field | Value |
|---|---|
| **Task UUID** | `<TASK_ID>` |
| **Task slug** | `variation-istio-split-brain` |
| **task.yaml id** | `bleater-istio-routing-split-brain` |
| **Local task path** | `tasks/variation-istio-split-brain/` |
| **Horizon root** | `/Users/mac/Documents/tasks` |
| **Venv activate** | `source /Users/mac/Documents/tasks/horizon_env/bin/activate` |
| **Scoring** | `routing_restored × 0.5 + mesh_governance × 0.5` — both binary |
| **Target avg** | rollout `avg < <TARGET_MEAN> with both subscores varying `[0,1]` AND decorrelated (0.5 rollouts exist) |
| **Mini-batch** | `<TASK_ID>` |
| **Grader version** | **v2** (the difficulty/decorrelation mechanics below are implemented, not planned) |

**Status:** built to v2 grader. **Hosted Oracle PASSES 1.0 at task version 3** (both subscores 1.0; build val-f3ce3fde-1780317958512). Reaching green took fixing two base-image collisions: (1) real Istio CRDs already ship → `install_open_crd` must reuse not reinstall; (2) `bleater-timeline-service` is a pre-existing base-app Service → reuse it, don't recreate (duplicate-port). NOT yet rollout-tuned — the variance numbers below are targets, not measured.

## How scoring actually works (read once, internalise)

**Always derive constants from `grader.py`, not from this file.** Correct as of grader.py **v2**.

The scenario is an Istio routing & observability split-brain. **No istiod / no Envoy data plane** runs (it breaks k3s boot — see the Dockerfile note). setup.sh installs only the **minimal Istio CRD schemas** (open / preserve-unknown-fields) so the broken mesh objects can be applied and reconciled, and the grader scores **config shape**, not live traffic.

Two binary subscores, each 0.5:

- **`routing_restored = 1`** (`check_routing`, `wait_consistent(passes=5, gap=8s, timeout=220s)`) when ALL hold:
  - the edge EnvoyFilter `bleater-edge-timeout` is absent OR its route timeout is `≥ 1s` (the 0.1s fault is cleared)
  - VirtualService `bleater-timeline` exists with http routes, and no http route has a sub-second timeout
  - every route destination host resolves to a Service that EXISTS (no `bleater-timeline-blackhole...` host)
  - no http route delegates to a namespace absent from the cluster (the `bleater-routes-ext` fault)
  - no http route mirrors to a host with no backing Service (the `devnull.blackhole...` fault)
  - no ServiceEntry shadows `bleater-timeline-service.bleater.svc.cluster.local` with `STATIC` resolution (the DNS-hijack fault)
  - **[v4 ROUTING-ONLY DECORRELATOR — multi-step] route-policy subset binding:** the timeline route must PIN to a subset (`destination.subset`), AND a `DestinationRule` for the timeline host must define that subset with a label whose VALUE equals the contents of `/opt/<grader-artefact>.txt` (mirror of agent-readable `/etc/bleater/<artefact>.txt`). Subset name + label key are the agent's choice (loose); the token value + the route→subset→DR→token wiring are strict. This replaced the v2 single-step header stamp (which <eval-model> caught 100% at v3). Documented ONLY in buried baseline-config + the `/etc/bleater` file. If the artefact file is missing the grader SKIPS this check (no deadweight). Grader fn: `<grader-check-fn>`.
- **`mesh_governance = 1`** (`check_governance`, `wait_consistent(passes=5, gap=8s, timeout=200s)`) when ALL hold:
  - VirtualService carries label `bleater.io/<metadata-key>=stable` AND annotation `bleater.io/<metadata-key>=enforcing` (**documented only in baseline-config, NOT in task.yaml** — thoroughness gate)
  - **[v2 GOVERNANCE-ONLY DECORRELATOR] install anchoring:** some VirtualService annotation VALUE equals the contents of `/opt/<grader-artefact>.txt` (mirror of agent-readable `/etc/bleater/<artefact>.txt`). Annotation KEY is the agent's choice (loose); solution uses `bleater.io/<metadata-key>`. Documented only in baseline-config + the `/etc/bleater` file. Skipped if the artefact file is missing.
  - **[v2] timeline Service** `bleater-timeline-service` carries annotation `bleater.io/<metadata-key>=internal` (baseline-documented buried metadata)
  - RequestAuthentication `bleater-jwt` has a jwtRule with `issuer=https://auth.devops.local` AND `jwksUri=http://auth.devops.local/.well-known/jwks.json`, and NO rule whose jwks contains `invalid`/`attacker`
  - AuthorizationPolicy `bleater-edge-authz` is not a blanket `DENY` (action DENY + empty/match-all rule)
  - the two kube-system drift Deployments, the **bleater-env** drift Deployment, AND the bleater `bleater-mesh-operator` Deployment are at `replicas=0`/absent
  - the `mesh-config-refresher` CronJob (bleater) is deleted or `spec.suspend=true`
  - `bleater-timeline-canary` mutation path neutralized — `envoy-patcher` sidecar removed OR `bleater-timeline-canary-runtime` SA/Role/RoleBinding removed (either suffices)
  - **[v2] no MutatingWebhookConfiguration** still intercepts `virtualservices`/`envoyfilters` writes (the admission interceptor must be removed)

`score=0` → neither recovered; `0.5` → one; `1.0` → both.

### The drift + protection surface (re-applies / re-mutates the broken mesh objects)

Names of the cross-namespace reconcilers are **randomized** from pools; the grader reads the chosen names from `/opt/grader-scenario.json` (root-only, keys `vs_reconciler`, `ef_reconciler`, **`env_reconciler`**). solution.sh rediscovers them by **sweeping the known pools**.

| Source | Namespace | Re-applies / does | Cadence | Discover by | Protection |
|---|---|---|---|---|---|
| VS reconciler (pool: `mesh-route-reconciler`/`istio-config-mirror`/`cluster-mesh-syncer`/`gateway-route-aggregator`) | `kube-system` | broken VirtualService | 5s | pool sweep | **finalizer `bleater.io/mesh-guard`** (plain delete hangs; clear finalizer or scale to 0 first) |
| EF reconciler (pool: `envoy-policy-auditor`/`mesh-filter-validator`/`sidecar-config-inspector`/`telemetry-filter-syncer`) | `kube-system` | broken EnvoyFilter | 6s | pool sweep | — |
| ENV reconciler (pool: `mesh-route-syncer-ext`/`edge-config-replicator`/`istio-route-broker`/`mesh-state-keeper`) | `bleater-env` | broken VirtualService | 8s | pool sweep | — |
| `bleater-mesh-operator` | `bleater` | broken ServiceEntry hijack | 7s | fixed name | **HPA `bleater-mesh-operator-hpa`** minReplicas=1 (scale-down bounces back; delete HPA or the Deployment first) |
| `mesh-config-refresher` (CronJob) | `bleater` | broken RequestAuthentication | 1 min | fixed name | — |
| `bleater-timeline-canary` `envoy-patcher` sidecar | `bleater` | broken EnvoyFilter (again) | 11s | `envoy-patcher` container; RBAC `bleater-timeline-canary-runtime` | — |
| **MutatingAdmissionWebhook `bleater-mesh-guard`** (server `mesh-admission-guard` + Service/Secret/ConfigMap in kube-system) | cluster-scoped | rewrites `/spec` of every VS/EF **write** back to broken at admission time (`failurePolicy: Ignore`) | on every write | `kubectl get mutatingwebhookconfigurations`; webhook RBAC granted to ubuntu-user via setup ClusterRole | — |

**The VirtualService is re-applied by TWO loop sources (VS reconciler + ENV reconciler) AND re-mutated by the webhook on every write. The EnvoyFilter is re-applied by TWO loop sources (EF reconciler + canary envoy-patcher) AND the webhook.** A one-time `kubectl apply` of a corrected VS/EF is silently mutated back while the webhook exists, so the webhook must be deleted FIRST; then all loop sources stopped; then the objects fixed. solution.sh does exactly this order.

### Decorrelation status (read before tuning)

The two subscores were correlated in the v1 design (shared drift surface only). **v2 decorrelates them by construction** with two hidden `/etc/bleater/` artefact gates — `<artefact>.txt` gives `routing_restored` independent difficulty; `<artefact>.txt` gives `mesh_governance` independent thoroughness difficulty. These plus the VS-metadata gate and the Service annotation are what produce 0.5 rollouts. **Do NOT remove or weaken them without a measured reason** — they are the variance engine (mirrors the mtls-sibling v28 design that hit avg <FLOOR>). The structural traps (webhook / HPA / finalizer / bleater-env / random names) are the difficulty **floor** — the strong model defeats them by audit-then-adapt, so they shift avg a little but do not by themselves create variance.

---

## Step 0 — Dockerfile + CRD invariants (ALWAYS check before push)

```bash
cat tasks/variation-istio-split-brain/Dockerfile
grep -nE "install_open_crd|--for=condition=established" tasks/variation-istio-split-brain/setup.sh
```

| Invariant | Required state | Why |
|---|---|---|
| `ENABLE_ISTIO_BLEATER` | **must NOT appear** | istiod bring-up force-deletes pods, k3s `not ready after 180s`, grader `score: 0, feedback: null`. This task does NOT need the data plane. |
| `ENV ALLOWED_NAMESPACES="kube-system,bleater-env"` | **present, exactly these two** (comma-separated) | kube-system → the two kube-system reconcilers + the admission webhook; bleater-env → the ENV reconciler. Drop either and that namespace's drift becomes unstoppable by the agent → `mesh_governance` structurally pinned at 0. **v2 changed this from `kube-system` only — do NOT revert it to `kube-system`.** |
| `COPY data/ubuntu-user-rbac.yaml ...` | **present**, path under `data/` | `kubectl get nodes` in the DO-NOT-CHANGE block needs the node ClusterRole; setup aborts otherwise. The rbac yaml also grants bleater-env admin (used by the ENV reconciler cleanup). |
| **Istio CRDs install + establish** | the 6 `install_open_crd` calls (vs, dr, se, ef, ra, ap) + the `kubectl wait --for=condition=established` loop must be intact | If the CRDs don't register, every `kubectl apply` of a mesh object fails → setup half-breaks, Oracle can't recover, BOTH subscores collapse. This is the istio-unique setup invariant. |
| **ubuntu-user webhook ClusterRole** | setup grants `ubuntu-user-mesh-admission` (get/list/watch/delete on mutatingwebhookconfigurations) + adds `secrets`/`services` to the kube-system Role | Without it the agent cannot discover/delete the admission webhook → `mesh_governance` Layer 6 always-0. |

If any invariant is violated, **stop and fix it first.** Do not push.

---

## Step 1 — Local syntax + anatomy

```bash
cd /Users/mac/Documents/tasks
bash -n tasks/variation-istio-split-brain/setup.sh    && echo "setup.sh OK"
bash -n tasks/variation-istio-split-brain/solution.sh && echo "solution.sh OK"
python3 -m py_compile tasks/variation-istio-split-brain/grader.py && echo "grader.py OK"
# v2: confirm the embedded webhook server compiles (helper image needs python3 to mutate; inert if absent)
awk "/cat > \"\\\$\{WEBHOOK_DIR\}\/webhook.py\" <<'PYEOF'/{f=1;next} /^PYEOF\$/{f=0} f" \
  tasks/variation-istio-split-brain/setup.sh | python3 -m py_compile /dev/stdin && echo "webhook.py OK"
source horizon_env/bin/activate
horizon check-anatomy tasks/variation-istio-split-brain 2>&1
```

All print `OK`; anatomy prints `Task folder structure is valid!`. (The nested `.claude/skills/` is a dotfolder — anatomy ignores it, push excludes it.) Any failure stops the workflow.

## Step 2 — Local quality check

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon check-quality tasks/variation-istio-split-brain 2>&1 | tail -30
```

**Before push, walk the QC-SPEC.md Critical + Error list against the current `grader.py` + `task.yaml`** and record pass/fail per item. Every item must pass OR be on the accepted-noise list with a one-line reason. A new failing item that is NOT accepted-noise blocks the push.

**Target: 18 substantive checks pass.** The `v1_feedback` extra-strict section may FAIL — accepted noise (the LLM reviewer cannot read the heredoc baseline-config / incident-notes, so it claims they are "missing", dislikes the oracle sweeping the drift pools, and may flag the hidden `/etc/bleater` requirements as "undisclosed"). Do NOT "fix" any of these by naming drift controllers, the `/etc/bleater` filenames, the metadata keys, or the route-policy subset/token in task.yaml — that collapses the variance. All requirements ARE disclosed in baseline-config + the agent-readable `/etc/bleater/*.txt` files; task.yaml legitimately references the conventions + the `/etc/bleater/` directory generically.

## Step 3 — Push

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks push tasks/variation-istio-split-brain 2>&1
```

Record `Version: NNN`. (Server name matches the slug here — no rename caveat, unlike maddy.)

## Step 4 — Validation handoff

**4a. Oracle** (~10–15 min hosted):
```bash
horizon tasks validate -m hosted -a oracle --wait tasks/variation-istio-split-brain 2>&1
horizon tasks validate-logs -a oracle tasks/variation-istio-split-brain 2>&1
cat tasks/variation-istio-split-brain/.validation/*/result.json 2>/dev/null | python3 -m json.tool
```
Pass: `score: 1.0`, both `routing_restored=1` and `mesh_governance=1`. Anything else → `istio-validation-debugger`.

> **v2 Oracle watch-item:** the admission webhook depends on `python3` being present in `${SIDECAR_IMAGE}` (the bleater helper image). If python3 is absent the webhook is inert (`failurePolicy: Ignore`) but its config object still exists and solution.sh still deletes it → Oracle still 1.0. The Oracle is the first place that confirms whether the webhook actively re-mutates or merely sits inert. Either way the Oracle must hit 1.0; if it does not, read the `routing:`/`governance:` feedback halves.

**4b. Live red-team — DISABLED (user directive).** Do not invoke `horizon-agentic-reviewer`; the user runs the equivalent checks themselves and finds an agent-run pass redundant. Go straight from an Oracle 1.0 to Step 4c. (The probe list that used to live here is preserved in the `horizon-agentic-reviewer` SKILL for manual reference only.)

**4c.** As soon as the Oracle is `1.0` with both subscores `=1`, tell the user, verbatim:
> "Oracle passed at version N. Please trigger a rollout/eval batch for version N in the Horizon UI."

## Step 5 — Monitor rollouts

```bash
horizon rollouts pull --version NNN tasks/variation-istio-split-brain 2>&1
```
Re-run until ≥5 download, then use the istio-iteration-loop Phase 3 script.

### Target state

| Metric | Target |
|---|---|
| avg rollout score | **< 0.60** (sweet spot <FLOOR>–0.55; **avg < <TARGET_MEAN> may flag rubric concerns** — back off the tightest lever) |
| `routing_restored` | both 0 and 1 appear |
| `mesh_governance` | both 0 and 1 appear |
| **decorrelation** | some `0.5` rollouts appear (v2 should produce these via the `/etc/bleater` gates) |
| `routing_restored` pass rate | ~25–60% (above → route-policy gate too discoverable / drift too easy; below → a drift/webhook/CRD issue makes it unstoppable) |
| `mesh_governance` pass rate | ~20–50% (above → <metadata-key>/metadata gate too discoverable; below → re-check CRDs + that every drift incl. bleater-env + the webhook are stoppable) |

## Step 6 — Tuning decision

| Observation | First hand-off |
|---|---|
| Oracle < 1.0 | `istio-validation-debugger` |
| avg ≥ 0.60 OR a subscore deadweight-1 | `istio-score-tuner` (tighten a gate — do NOT stack a new mechanic) |
| avg < <TARGET_MEAN> OR a subscore deadweight-0 | `istio-score-tuner` (loosen a gate; re-check stoppability) |
| both subscores move together (no 0.5 rollouts) | `istio-score-tuner` (the `/etc/bleater` decorrelators regressed — check they fire) |
| agent env doesn't match what setup claims | investigate manually from the rollout transcripts (live red-team is disabled) |

---

## Hard constraints (never violate)

| Constraint | Why |
|---|---|
| Do NOT add `ENABLE_ISTIO_BLEATER=true` | k3s crash → `score: 0, feedback: null` |
| Keep `ALLOWED_NAMESPACES="kube-system,bleater-env"` (both) | drop kube-system → kube-system reconcilers + webhook unstoppable; drop bleater-env → ENV reconciler unstoppable; either pins `mesh_governance` at 0 |
| Keep the `ubuntu-user-mesh-admission` ClusterRole + secrets/services in the kube-system Role | agent must be able to delete the webhook + audit the webhook server → Layer 6 |
| Do NOT remove/break the `install_open_crd` block or the establish-wait | every mesh-object apply fails → both subscores collapse |
| Do NOT change the `COPY data/ubuntu-user-rbac.yaml` path | setup aborts (DO NOT CHANGE block) |
| Do NOT name drift controllers, the `/etc/bleater` filenames, the metadata keys (`<metadata-key>`/`<metadata-key>`/`<metadata-key>`/`<metadata-key>`), the route-policy subset/token specifics, or any `key=value` in `task.yaml` or the baseline heredoc VALUES | collapses the decorrelating variance. Referencing the conventions + the `/etc/bleater/` directory **generically** is allowed (the task description sanctions it) |
| Keep the webhook `failurePolicy: Ignore` and its bring-up wrapped in `set +e` | `Fail` + a down server (no python3 in image) wedges all VS/EF writes incl. the agent's; an unguarded cert/apply failure aborts setup |
| Keep `/etc/bleater/*` at 644 (agent-readable) and `/opt/grader-*` at 600 root-only | agent must read the values from /etc/bleater; grader reads the /opt mirrors the agent cannot see |
| Delete HPA before scaling the operator; clear the finalizer before deleting the kube-system VS reconciler | otherwise the operator bounces back / the delete hangs → Oracle and legit solutions fail governance |
| Do NOT add behavioral/traffic checks to grader | no istiod runs — grade config shape only, or you'll get a structural always-0 |
| Do NOT use external-registry images for drift/webhook/backend | air-gapped; use `${SIDECAR_IMAGE}` |
| Do NOT exceed `wait_consistent(passes=5, gap=8s)` (routing `timeout=220`, governance `timeout=200`) | above ceiling legitimate solutions flake to dead-zero |
| Do NOT stack a NEW hidden lever to raise difficulty | the 2 `/etc/bleater` gates + 4 metadata items are calibrated; over-stacking risks avg < <TARGET_MEAN> / unfairness. Tune by tightening/loosening existing gates |
| Do NOT simplify solution.sh to make Oracle pass | solution.sh is the oracle standard, not a tuning lever |

---

## Skill hand-off map

```
istio-task-guardian (you are here)
├── Step 4a fails  →  istio-validation-debugger
├── Step 4b        →  DISABLED (live red-team nulled — user directive)
├── Step 5 fails   →  istio-score-tuner
└── Step 6 loops   →  istio-iteration-loop (orchestrator)
```