# Istio Validation Debugger

> Fetches and interprets validation logs (No-Op and Oracle) for the variation-istio-split-brain task (<TASK_ID>). Maps observed log patterns to root causes and the smallest safe fix. Use after a validation fails to understand why it failed before editing anything.

- Skill: `omonuj/istio-validation-debugger` (Agent Skill)
- Install (CLI): `npx skillmds@latest add omonuj/istio-validation-debugger`
- Raw SKILL.md: https://api.skillmd.com/api/skills/omonuj/istio-validation-debugger/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: omonuj (https://skillmd.com/u/omonuj)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/omonuj/istio-validation-debugger

---


# istio-validation-debugger

## Execution Mode — FULLY AUTONOMOUS

**NEVER ask the user for confirmation.** Fetch logs, interpret, propose and apply fixes immediately. The only user-facing handoff in the loop is triggering rollouts (owned by `istio-iteration-loop`). This skill never has user-facing handoffs.

## Goal

Turn an Oracle or No-Op failure into a specific root cause with a specific fix, from log evidence — not guesswork. If you cannot map the pattern to a row below with confidence, investigate manually from the `output.txt` setup log + a re-read of setup.sh/solution.sh/grader.py before changing any file. (The live red-team / `horizon-agentic-reviewer` is DISABLED for this task — do not invoke it.)

## Trigger

Oracle ≠ `1.0/passed`; No-Op `score > 0` (false positive); grader exception; `feedback: null` (setup crash); or you are about to edit setup/solution/grader after a validation failure. NOT for rollout failures — use `istio-score-tuner`.

## Task Identity

| Field | Value |
|---|---|
| **Task UUID** | `<TASK_ID>` |
| **Slug / path** | `variation-istio-split-brain` / `tasks/variation-istio-split-brain/` |
| **Subscores** | `routing_restored` (0.5) + `mesh_governance` (0.5), both binary |
| **Venv** | `source /Users/mac/Documents/tasks/horizon_env/bin/activate` |
| **Grader version** | **v2** — includes the admission webhook, the bleater-env drift, the HPA/finalizer protections, and the two `/etc/bleater` decorrelating gates |

**v2 root-only files the grader reads:** `/opt/grader-scenario.json` (keys `vs_reconciler`, `ef_reconciler`, **`env_reconciler`**), `/opt/<grader-artefact>.txt`, `/opt/<grader-artefact>.txt`. Agent-readable mirrors: `/etc/bleater/<artefact>.txt`, `/etc/bleater/<artefact>.txt`. If a `/opt/grader-*.txt` artefact is missing the grader SKIPS that gate (no deadweight) — so a "governance/routing always-1" with the value-gate silently skipped means the setup write failed; check `output.txt`.

Grader `feedback` is `routing: <detail>; governance: <detail>` — read both halves; each names the layer that fired.

## Result shapes

| score | passed | feedback | meaning |
|---|---|---|---|
| 0 | false | null | container/setup crash before grader ran → `output.txt` |
| 0 | false | non-null | both subscores failed (grader ran) |
| 0.5 | false | non-null | one passed, one failed |
| 1.0 | true | non-null | both passed |

For No-Op, anything but `score: 0` is a problem.

## Step 1 — Fetch + read result

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks validate-logs -a oracle tasks/variation-istio-split-brain 2>&1
cat tasks/variation-istio-split-brain/.validation/*/result.json 2>/dev/null | python3 -m json.tool
```
Logs: `tasks/variation-istio-split-brain/.validation/<build_id>/{output.txt,result.json}`.

## Step 2 — Map symptom → cause → fix

### Branch A. `feedback: null` / `score: 0` — setup crash (istio-specific causes first)

| Pattern in `output.txt` | Root cause | Fix |
|---|---|---|
| `error: unable to recognize ... no matches for kind "VirtualService"` / `EnvoyFilter` etc. | The minimal Istio CRDs didn't install or didn't establish before the CR applies | Confirm the 6 `install_open_crd` calls run AND the `kubectl wait --for=condition=established` loop precedes the first mesh-object apply. The faults apply too early if the wait is missing. |
| `timed out waiting for the condition` on a crd | establish-wait too short or apiserver slow | The wait is `--timeout=60s` with `|| true`; if a CR apply still races it, add a short `sleep 3` after the establish loop |
| `k3s is not ready after 180 seconds` | `ENABLE_ISTIO_BLEATER=true` in Dockerfile | Remove it entirely |
| `Error from server (Forbidden)` on `kubectl get nodes` | rbac yaml not copied / wrong path | Restore `COPY data/ubuntu-user-rbac.yaml ...` |
| `failed to pull image` for a drift/backend/webhook pod | external registry referenced | Use `${SIDECAR_IMAGE}` (air-gapped) |
| setup aborts during the webhook/cert/HPA block | a command failed under `set -e` | The webhook block is wrapped in `set +e ... set -e` and the HPA apply ends in `|| true` — if setup still aborts there, a guard was removed; restore it. Webhook/HPA are best-effort floor; they must NEVER abort setup |
| webhook pod `CrashLoopBackOff` / logs `python3 unavailable; webhook inert` | no python3 in `${SIDECAR_IMAGE}` | Expected-tolerable: `failurePolicy: Ignore` means writes pass unmutated; the webhook is inert but its config still must be removed (Layer 6 still gates). Not a setup failure |
| `setup.sh: line N: syntax error` | bash error | `bash -n setup.sh` |

### Branch B. No-Op `score > 0` — false positive

| Subscore passed on No-Op | Why wrong | Fix |
|---|---|---|
| `routing_restored=1` | The faults didn't apply (CRD race) OR drift didn't deploy so the broken objects were never (re)written OR the route-policy gate silently SKIPPED (`/opt/<grader-artefact>.txt` missing) | Confirm the broken VS/EF/SE applied (Branch A CRD check), the reconcilers exist, and `/opt/<grader-artefact>.txt` was written (the `/etc/bleater` block ran) |
| `mesh_governance=1` | Drift didn't deploy, OR the broken RA/authz weren't injected, OR the VS already has <metadata-key>/<metadata-key> at setup time, OR the <metadata-key> gate silently SKIPPED (`/opt/<grader-artefact>.txt` missing), OR the broken VS already happens to satisfy a gate | Confirm setup sets the wrong VS metadata (`<metadata-key>=experimental`, no <metadata-key>/<metadata-key>), Service has NO `<metadata-key>`, attacker JWKS, blanket-deny authz, all FOUR drift deploys (incl. bleater-env) + the webhook deployed, and `/opt/<grader-artefact>.txt` was written |

### Branch C. Oracle `0.5` — `routing_restored=0` (governance passed)

Read the `routing:` half of feedback:

| Feedback fragment | Cause | Fix in solution.sh |
|---|---|---|
| `edge EnvoyFilter ... still clamps route timeout` | EnvoyFilter re-applied mid-window | Stop BOTH re-appliers before deleting it: the EF kube-system reconciler (sweep `EF_RECONCILER_POOL`) AND the canary `envoy-patcher` (delete the canary). Then `kubectl delete envoyfilter bleater-edge-timeout -n bleater` |
| `routes to host ... with no backing Service` (blackhole) | VS reconciler re-applied the broken VS | Stop the VS kube-system reconciler (sweep `VS_RECONCILER_POOL`) before re-applying the corrected VS pointing at `bleater-timeline-service` |
| `delegates to namespace ... that does not exist` | corrected VS still has the `bleater-routes-ext` delegate, OR the **webhook** re-mutated the VS, OR the **ENV reconciler** (bleater-env) re-applied the broken VS | Re-apply VS without the delegate route; **delete the `bleater-mesh-guard` MutatingWebhookConfiguration FIRST**; stop the ENV reconciler (sweep `ENV_RECONCILER_POOL` in bleater-env) AND the VS reconciler |
| `mirrors traffic to ... blackhole sink` | corrected VS still has the devnull mirror, OR webhook/ENV/VS reconciler re-broke it | Re-apply VS without `mirror`/`mirrorPercentage`; ensure webhook deleted + both VS re-appliers stopped |
| `ServiceEntry ... hijacks in-cluster hostname` | `bleater-mesh-operator` re-applied the SE | Stop `bleater-mesh-operator` (bleater) — **delete its HPA `bleater-mesh-operator-hpa` first** (or it bounces back) — before `kubectl delete serviceentry bleater-timeline-resolution -n bleater` |
| `does not bind the platform route-policy token from /etc/bleater/<artefact>.txt to its route via a DestinationRule subset` | **[v4 routing-only multi-step gate]** route isn't pinned to a subset, or no DestinationRule defines that subset with the token label | Read `/etc/bleater/<artefact>.txt`; create a DestinationRule on the timeline host with a subset labelled `<anykey>=<token>` AND set the VS primary route `destination.subset` to that subset name. If this is the ONLY routing failure and Oracle should be 1.0, solution.sh is missing the DR or the subset wiring (check Step 8/8b) |

### Branch D. Oracle `0.5` — `mesh_governance=0` (routing passed)

| Feedback fragment | Cause | Fix in solution.sh |
|---|---|---|
| `VirtualService missing required label <metadata-key>=stable` / `annotation <metadata-key>=enforcing` | corrected VS lacks the baseline metadata | Add label `bleater.io/<metadata-key>: stable` + annotation `bleater.io/<metadata-key>: enforcing` to the VS |
| `does not anchor to the current platform install id from /etc/bleater/<artefact>.txt` | **[v2 governance-only gate]** VS has no annotation whose value == the install id | Read `/etc/bleater/<artefact>.txt`; add an annotation with that value (solution uses `bleater.io/<metadata-key>`). If this is the only governance failure and Oracle should be 1.0, solution.sh dropped the annotation |
| `timeline Service missing required annotation <metadata-key>=internal` | **[v2]** Service not annotated | `kubectl annotate service bleater-timeline-service -n bleater bleater.io/<metadata-key>=internal --overwrite` |
| `bleater-jwt ... does not validate against documented issuer/jwks` or `still trusts attacker JWKS` | RA not reconciled | Apply RA with `issuer=https://auth.devops.local`, `jwksUri=http://auth.devops.local/.well-known/jwks.json` |
| `AuthorizationPolicy ... is a blanket DENY` | authz still deny-all | Delete `bleater-edge-authz` or set action ALLOW |
| `active drift ... <name>@kube-system` | a kube-system reconciler still running (the VS-pool one carries a **finalizer** — plain delete hangs) | Sweep BOTH pools; **clear finalizers (`kubectl patch ... -p '{"metadata":{"finalizers":[]}}'`) before delete**, or scale to 0 |
| `active drift ... <name>@bleater-env` | **[v2]** the ENV reconciler still running | Sweep `ENV_RECONCILER_POOL` in bleater-env; delete the bleater-side Role/RoleBinding + bleater-env SA. Needs `ALLOWED_NAMESPACES` to include bleater-env |
| `active drift ... bleater-mesh-operator` | local operator running, **HPA bounces it back to 1** | Delete HPA `bleater-mesh-operator-hpa` FIRST, then delete the deployment |
| `cronjob mesh-config-refresher still active` | refresher not stopped | delete or `spec.suspend=true` |
| `bleater-timeline-canary mutation capability not fully neutralized` | both sidecar AND runtime RBAC present | delete the canary deployment OR remove `bleater-timeline-canary-runtime` SA/Role/RoleBinding |
| `MutatingWebhookConfiguration ... still intercepts mesh-object writes` | **[v2]** the admission webhook config not removed | `kubectl delete mutatingwebhookconfiguration bleater-mesh-guard` (solution does this FIRST, before any VS/EF apply, so the fix isn't re-mutated). Needs the `ubuntu-user-mesh-admission` ClusterRole |

### Branch E. Oracle `0` non-null — both failed
Read both halves; follow Branch C and Branch D in parallel. If it stays 0 after fixing feedback, re-run No-Op to isolate a setup-side issue.

### Branch F. Grader Python exception
`json.JSONDecodeError`/`KeyError`/`TimeoutExpired` — all guarded by `kubectl_json`/`run_kubectl`/`.get`. If one crept past, wrap it; treat failure as subscore=0. Not in the hard-constraint list — fix in place.

---

## Step 3 — Pre-fix safety
1. Dockerfile + CRD invariants hold (guardian Step 0).
2. `bash -n setup.sh && bash -n solution.sh && python3 -m py_compile grader.py`.
3. If it touches subscore difficulty → diagnose with `istio-score-tuner` first.
4. Articulate the why in one sentence.

| Fix type | Hand off to |
|---|---|
| setup.sh fault-injection bug | `istio-task-guardian` Step 3 (push) |
| solution.sh bug (Branch C/D) | `istio-task-guardian` Step 3 |
| grader exception (Branch F) | `istio-task-guardian` Step 3 |
| subscore difficulty | `istio-score-tuner` first |
| unmappable | manual investigation (read `output.txt` + setup.sh/solution.sh/grader.py); live red-team is disabled |

## Known non-issues (do NOT fix)

| Pattern | Why |
|---|---|
| `v1_feedback` section FAILs | reviewer can't read heredoc baseline-config; dislikes pool-sweep oracle. Accepted |
| No-Op `score: 0` with feedback | grader working correctly |
| governance fails because agents only audited `bleater` | the kube-system discovery axis working as intended |
| `behavior_in_tests` complaint (graded config not traffic) | correct by design — no istiod; grade shape |

## Verification cheat sheet

| Hypothesis | Probe |
|---|---|
| "CRDs install + establish before CR apply" | `grep -n "install_open_crd\|--for=condition=established" setup.sh` — establish loop must precede the `kubectl apply -f` of the broken objects |
| "solution deletes the webhook FIRST, before any VS/EF apply" | read solution.sh — `delete mutatingwebhookconfiguration` must precede the VS/EF deletes and the corrected VS apply (else the apply is re-mutated) |
| "solution stops EF's TWO re-appliers before deleting EF" | read solution.sh — EF reconciler sweep + canary delete must precede `delete envoyfilter` |
| "solution stops VS's TWO re-appliers" | read solution.sh — both VS reconciler pool sweep AND `ENV_RECONCILER_POOL` (bleater-env) sweep present |
| "solution sweeps all three pools" | `grep -E "VS_RECONCILER_POOL\|EF_RECONCILER_POOL\|ENV_RECONCILER_POOL" solution.sh` |
| "solution clears finalizers + deletes HPA before the operator" | `grep -nE "finalizers|hpa|bleater-mesh-operator-hpa" solution.sh` — HPA delete before operator delete; finalizer patch before kube-system deletes |
| "solution reflects the two /etc/bleater artefacts" | `grep -nE "<artefact>.txt\|<artefact>.txt\|<metadata-key>\|x-bleater-route-policy" solution.sh` |
| "grader reads root-only mirrors, not /etc/bleater" | `grep -nE "/opt/grader-(install-id\|route-policy).txt\|env_reconciler" grader.py` |
| "wait_consistent unchanged" | `grep -E "wait_consistent" grader.py` — routing `passes=5,gap=8,timeout=220`; governance `passes=5,gap=8,timeout=200` |
| "no behavioral check crept into grader" | `grep -iE "s_client\|curl\|openssl\|http_code\|connect" grader.py` — must return nothing (config-shape only) |
| "no specifics leaked to task.yaml" | `grep -iE "<metadata-key>\|<metadata-key>\|<metadata-key>\|<metadata-key>\|x-bleater-route-policy\|install-id\|route-policy\|kube-system\|bleater-env\|reconciler\|mesh-guard" task.yaml` — must return nothing |