# Istio Score Tuner > Diagnose rollout regressions and tune difficulty for the variation-istio-split-brain task (). Specializes in keeping avg rollout score < with both subscores varying AND decorrelated, by tightening/loosening the two implemented /etc/bleater gates and the metadata gates — never by stacking new mechanics and never by adding behavioral/traffic checks (no istiod runs). Picks the smallest safe lever for the observed symptom. - Skill: `omonuj/istio-score-tuner` (Agent Skill) - Install (CLI): `npx skillmds@latest add omonuj/istio-score-tuner` - Raw SKILL.md: https://api.skillmd.com/api/skills/omonuj/istio-score-tuner/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Productivity - Author: omonuj (https://skillmd.com/u/omonuj) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/omonuj/istio-score-tuner --- # istio-score-tuner ## QC Spec compliance — assert before every tuning edit Any difficulty/variance lever you pull must STILL satisfy the QC Spec (`.claude/QC-SPEC.md`). Before recommending a change, assert it does not breach: - **No Binary Bundling** — never recover variance by AND-ing more heterogeneous checks into one subscore. - **Incremental Rewards / No Dead Weights** — no subscore left always-0 or always-1; every lever keeps both subscores varying. - **Functional Subscore Variance** — at least one functional subscore takes ≥2 distinct values across a batch. - **Equal Weights** — never rebalance weights to hit a number. - **Functional Tests as Subscores / No Answer Leakage** — don't convert a functional subscore into static matching, and don't leak the answer to manufacture passes. A lever that lowers avg by breaching any of these is NOT a valid fix — pick a different lever. ## Execution Mode — FULLY AUTONOMOUS **NEVER call `ask_user` or pause for confirmation.** Pick the best path and execute it immediately. ## Goal Keep avg rollout score `< ` (sweet spot –0.55) with both subscores varying, **decorrelated**, no deadweight. Every edit is the smallest reversible change that explains the symptom. Invoked when: rollouts regress (avg ≥ 0.60); a subscore is deadweight (always-0 or always-1); the two subscores move together every batch (correlation collapse); avg drops below (over-hard → rubric-concern risk); `istio-task-guardian` Step 6 hands off. NOT for Oracle failures — use `istio-validation-debugger`. ## Task Identity | Field | Value | |---|---| | **Task UUID** | `` | | **Slug** | `variation-istio-split-brain` | | **Scoring** | `routing_restored × 0.5 + mesh_governance × 0.5`, both binary | | **Target** | `avg < (and `> ~`), both in `{0,1}` somewhere, NOT perfectly correlated | | **Grader version** | **v2** — decorrelation is implemented (two `/etc/bleater` gates) | **Status / MEASURED DATA:** - **task v3 (grader v2, single-step gates): rollout avg = 1.000, both subscores deadweight-1, zero 0.5s** on a 5-batch of -max-nebula. Lesson confirmed: against this model, *single-step* hidden gates documented even semi-clearly are caught 100% ( = "set an annotation", route-policy = "set a header"). Burying + de-hinting alone (Layer A, ~40-60% catch on the sibling) was judged insufficient on its own. - **task v4 response:** (a) baseline-config rewritten as terse buried PROSE (no `--- section ---` checklist headers; gates live mid-paragraph); (b) task.yaml de-hinted (no gate list, no `/etc/bleater` callout, no "read in full / audit rejects functionally-correct routes"); (c) the routing gate was converted from a single-step header stamp into a **multi-step `DestinationRule`-subset binding** (the istio analog of the sibling's cert-OID heavy lever): the route must pin to a subset that a DestinationRule defines and tags with the route-policy token. Multiple botch modes (no DR / not pinned / token-not-in-labels / subset-name mismatch). Oracle stays 1.0 (mock-harness verified). **Awaiting v4 rollout to measure.** - The governance lever () stays single-step but is now buried (Layer A). If v4 still shows governance deadweight-1, give governance its own multi-step lever too (e.g. require the install-id under a subset/DR label as well, or a second coordinated object). - **task v4 (multi-step subset + burial + de-hint): rollout avg = 1.000 AGAIN, deadweight-1, 0× 0.5** on a 10-batch (8 valid). Transcript proof: `cat`s the buried baseline IN FULL, `cat`s `/etc/bleater/*`, builds the DestinationRule + subset, sets , wires the VS — flawlessly, even offering to label the real pods. **Hard lesson: against , NO amount of burial / de-hinting / declarative "multi-step" config creates variance** — it reads every doc, audits all state (`kubectl get -A -o yaml`), and writes correct YAML every time. The sibling's variance came from an EXECUTION pipeline (openssl cert regen) this config-shape task lacks. - **task v5 response — add a real derivation/execution step:** both hidden gates now require the **SHA-256 FINGERPRINT of the token** (stripped value), not the raw token. `read file → strip → hash → embed`. The natural lazy action `sha256sum /etc/bleater/` folds in the trailing newline → wrong digest → FAIL; only hashing the stripped value passes. Documented precisely in baseline (fair), Oracle-safe (solution `sha256sum` of `printf %s` == grader `hashlib.sha256(.strip())`, verified), and a genuine botch surface. Grader: ``/``. **Awaiting v5 rollout.** - **task v5 (SHA-256 fingerprint gates): rollout avg = 1.000 AGAIN, deadweight-1** (4 valid). handles a 1-step derivation (strips + hashes correctly). So even a derivation step isn't enough on its own. - **task v6 — genuine execution pipeline (the cert-OID analog):** RequestAuthentication must serve an INLINE `jwks` DERIVED from the platform RSA signing key (public half at `/etc/bleater/jwt-signing-pub.pem`): read key → extract modulus → base64url → embed JWK. NOT a jwksUri (issuer's endpoint is air-gapped/unreachable). Grader (`` + `EXPECTED_MODULUS` via openssl) compares the modulus INTEGER (loose on encoding, strict on the key). Oracle-safe: solution derives via `openssl rsa -modulus` + python base64url; verified end-to-end on a real key that solution's `n` decodes to the grader's expected modulus. Botch surface: kept `jwksUri` (no inline) / mangled `n` / wrong key. **Awaiting v6 rollout.** - **task v6 (crypto inline-JWKS execution): avg = 1.000 (4th straight).** CONFIRMED: every config-shape lever (burial, de-hint, multi-step declarative, sha256 derivation, crypto execution) is caught by . Config shape cannot move this model. - **task v7 = FULL FUNCTIONAL+LIVE restructure (the maddy-proven approach; user-approved major change).** Added a live relay tier the agent must BUILD: `bleater-mesh-sink` (python3 recorder, infra) + broken `bleater-timeline-router` (the agent converges it to 2 replicas of a python3 relay that forwards `pod= route=mesh` to the sink AND serves per-pod `/metrics` :9090 with `mesh_route_delivered_total`/`mesh_route_attempts_total` INCREASING + POD_NAME downward API) + `bleater-timeline-router-legacy` (must remove) + a per-pod divergence recording rule. Grader execs into pods (``, ``) and requires per-pod delivery + metric counters to keep INCREASING across the window (Layer 7 routing; Layers 7-9 governance). Config-shape istio gates KEPT as floor. This is the ONLY mechanism shown to beat (maddy v4=0.70). Awaiting v7 rollout. - **Tuning the functional lever (once measured):** if avg too high → the relay is too easy to build right; tighten (require BOTH counters increasing, more replicas, stricter per-pod log format, faster cadence so a sloppy relay stalls the streak). If deadweight-0/Oracle flakes → the live timing is too tight; relax (lower wait_consistent passes for the functional layer, slow relay sleep, widen gap). NEVER revert to config-shape-only — it's proven not to vary. --- ## How difficulty is constructed (read once, internalise) The agent restores a corrupted Istio routing layer. **No istiod runs** — setup installs CRDs only and the grader scores **config shape**. The fix = revert the routing faults + restore auth + stop every drift/protection source + apply the baseline metadata + reflect the two hidden `/etc/bleater` artefacts. ### The shared surface (the floor — moves BOTH subscores) Drift + protection sources re-apply/re-mutate broken mesh objects; the cross-namespace ones are randomly named (pools), discovered by sweeping the pools: | Source | Namespace | Re-applies / does | Cadence | Protection | |---|---|---|---|---| | VS reconciler (pool) | kube-system | broken VirtualService | 5s | finalizer `bleater.io/mesh-guard` | | EF reconciler (pool) | kube-system | broken EnvoyFilter | 6s | — | | ENV reconciler (pool) | bleater-env | broken VirtualService | 8s | — | | `bleater-mesh-operator` | bleater | broken ServiceEntry | 7s | HPA `bleater-mesh-operator-hpa` (minReplicas=1) | | `mesh-config-refresher` (CronJob) | bleater | broken RequestAuthentication | 1m | — | | `bleater-timeline-canary` `envoy-patcher` | bleater | broken EnvoyFilter (again) | 11s | — | | MutatingWebhook `bleater-mesh-guard` | cluster | re-mutates VS/EF `/spec` on every write | per-write | `failurePolicy: Ignore` | Stopping this whole set helps **both** subscores (while active it re-breaks VS/EF/SE → `routing_restored` fails, and the deployments are governance-graded at `replicas=0` → `mesh_governance` fails). So shared-surface levers (cadence, pool obscurity, finalizer/HPA presence, webhook) **shift avg but do NOT decorrelate** — the strong model audits-then-adapts past them (mtls-sibling lesson). Use them only to move avg, never to create the 0.5 band. ### `routing_restored` independent difficulty — IMPLEMENTED (v2) `check_routing` (`wait_consistent(passes=5, gap=8s, timeout=220s)`): EF timeout ≥1s; VS primary route → real Service; no dead-ns delegate; no black-hole mirror; no hijacking ServiceEntry; **AND the routing-only decorrelator (``, multi-step as of v4):** the route must pin to a subset (`destination.subset`) AND a `DestinationRule` for the timeline host must define that subset with a label VALUE == `/opt/.txt` (agent reads `/etc/bleater/.txt`; subset name + label key loose). Documented only in buried baseline-config + the file. This gives routing independent, multi-step difficulty: an agent who stops all drift + reverts every fault still fails routing unless it reads the artefact AND wires route→subset→DR→token correctly (botch modes: no DR / not pinned / token-not-in-labels / subset-name mismatch). ### `mesh_governance` independent difficulty — IMPLEMENTED (v2) `check_governance` (`wait_consistent(passes=5, gap=8s, timeout=200s)`): VS `=stable` + `=enforcing`; **install anchoring** (some VS annotation VALUE == `/opt/.txt`; key loose); Service `bleater-timeline-service` annotation `=internal`; RA JWKS/issuer=baseline; authz not blanket-deny; all four drift Deployments (incl. **bleater-env**) `replicas=0`; refresher CronJob suspended; canary neutralized; **no MutatingWebhookConfiguration** intercepting VS/EF. The governance-only decorrelators are the gate + the // metadata (an agent who stops everything still fails governance if they skip these). ### The two `/etc/bleater` gates ARE the variance engine `.txt` (routing-only) and `.txt` (governance-only) are the decorrelators — each documented only in baseline-config + its `/etc/bleater` file, value mirrored root-only to `/opt/grader-*.txt`, loose on key/header so the failure is "didn't read the artefact," not "guessed the wrong key." They mirror the mtls v28 design (install-id + cert-OID) that produced avg with both subscores varying. **Treat them as load-bearing. Tighten/loosen them; do not delete them.** --- ## Four primary rules 1. **Preserve variance.** A subscore stuck at one value is broken. Always-0 → re-check CRD establishment, `ALLOWED_NAMESPACES=kube-system,bleater-env`, webhook stoppability. Always-1 → the matching `/etc/bleater`/metadata gate is too discoverable → tighten it. 2. **Don't re-invent decorrelation.** v2 already decorrelates via the two `/etc/bleater` gates. If 0.5 rollouts vanish, the gates regressed — verify they fire (grader reads `/opt/grader-*.txt`; setup writes both `/etc/bleater` and `/opt`) — do NOT pull a shared-drift lever. 3. **Functional > static, but config-shape only.** "Functional" here = config the agent must actively produce (a real annotation/header value sourced from `/etc/bleater/`, a corrected route), not a baseline-copyable constant. **Never add a traffic/handshake check** — structurally always-0 (no istiod). 4. **Minimal safe edits, never stack.** One file, one purpose, reversible. Do NOT add a NEW hidden lever to raise difficulty — over-stacking risks avg < / unfairness. Move an existing gate's strictness instead. --- ## Tuning order ### Level 1 — setup.sh + grader.py gate strictness (primary levers) **Decorrelating gate strictness (the main dials — move ONE subscore):** | Symptom | Lever | How | |---|---|---| | `mesh_governance` always-1 ( too easy) | tighten | require the value under a SPECIFIC annotation key (e.g. only `bleater.io/`) instead of any-annotation-value-match, in `check_governance` | | `mesh_governance` always-0 ( unfair) | loosen | also accept the value in a LABEL, not just annotations; confirm baseline names the `/etc/bleater/.txt` path; (current loose-on-key is already lenient) | | `routing_restored` always-1 (route-policy subset gate too easy) | tighten | require the subset on the PRIMARY route specifically; or require a second coordinated field; in `` | | `routing_restored` always-0 (subset gate unfair / unstoppable) | loosen / re-check | confirm buried baseline still describes the subset+DR mechanism + the `/etc/bleater` token; LOOSEN by also accepting the token stamped as a request header (revert toward the v2 single-step form) — that's the fastest fairness valve; confirm `/opt/.txt` exists; verify drift/webhook stoppable | | avg slightly high, both vary | drop the Service `` requirement OR keep — it's a mild governance lever | toggle the Layer-0b check | **Shared-surface levers (move BOTH — use only to shift avg, never to decorrelate):** | Lever | Where | Current | Raise difficulty | Floor | |---|---|---|---|---| | EF reconciler cadence | `emit_drift_deploy ... 6` | 6s | →5s | 5s | | `bleater-mesh-operator` cadence | `... 7` | 7s | →5s | 5s | | ENV reconciler cadence | `... 8` | 8s | →6s | 5s | | canary envoy-patcher cadence | sidecar `sleep 11` | 11s | →8s | 8s | | pool-name innocuousness | `*_RECONCILER_POOL` | innocuous | keep; must stay pool-sweepable by solution.sh | — | ### Level 2 — grader.py wait_consistent (when gates exhausted) | Lever | Current | Ceiling | |---|---|---| | routing `wait_consistent` | `passes=5, gap=8, timeout=220` | gap 8s; timeout floor 180s | | governance `wait_consistent` | `passes=5, gap=8, timeout=200` | gap 8s; timeout floor 150s | Never add a behavioral check. Never require an exact drift name (use pool sweep / `replicas=0`). Never require an exact webhook NAME (match by rule on virtualservices/envoyfilters, as the grader already does). ### Level 3 — task.yaml wording (last resort) Allowed: strengthen/weaken the "more than one namespace / audit /allowed_namespaces" hint; the "admission-layer interception" hint; the "protected drift (HPA/finalizer)" hint; sharpen/soften the generic "metadata conventions + /etc/bleater artefacts" phrasing. **Forbidden:** naming any drift controller, `kube-system`/`bleater-env`, the fault list, the `/etc/bleater` filenames, the exact metadata keys (``/``/``/``), the route-policy subset/token specifics, or itemizing the fix path. --- ## Diagnosis guide | Symptom | Root cause | Lever | |---|---|---| | No 0.5 rollouts (perfect correlation) | the `/etc/bleater` decorrelators are not firing | verify setup writes `/etc/bleater/*` AND `/opt/grader-*`; grader reads `/opt`; baseline documents both paths. Do NOT pull a shared-drift lever | | `routing_restored` always 1 | route-policy gate too discoverable / drift too easy | tighten route-policy (primary-route-only / specific key); speed up canary+EF cadence | | `routing_restored` always 0 (Oracle 1.0) | a drift source out-races the window, the webhook isn't being deleted by agents, OR CRDs didn't establish | verify CRD establishment; confirm webhook is discoverable+deletable (ClusterRole granted); if cadence < floor, raise slightly | | `mesh_governance` always 1 | / metadata gate too discoverable | tighten (specific key); keep // buried in baseline only | | `mesh_governance` always 0 (Oracle 1.0) | a gate's baseline wording is missing, OR a drift dep (incl. bleater-env / webhook / HPA / finalizer) is unstoppable | confirm baseline documents every gate; confirm all pools sweepable + webhook deletable + HPA/finalizer addressable | | avg 0.60–0.70 | slightly easy | tighten ONE decorrelating gate, not a shared one | | avg < | over-hard (rubric-concern risk) | loosen the tightest gate (e.g. accept install-id as label-or-annotation); slow the fastest cadence | --- ## Specialized roles - **Role 1 Rollout Analyst** — run the iteration-loop Phase 3 script, read ≥2 transcripts (1 pass, 1 fail), classify (deadweight / healthy / correlated / over-hard). Output one paragraph; no fixes. - **Role 2 Setup Drift Specialist** — which of the 6 loop sources + webhook + HPA + finalizer do failing agents miss? (The canary envoy-patcher is the sneaky 2nd EF re-applier; the ENV reconciler is the bleater-env one agents who only audit kube-system miss; the webhook silently re-mutates one-time applies.) Output ranked impact. - **Role 3 Routing Specialist** — `routing_restored` independence today = the route-policy `/etc/bleater` gate + stopping every VS/EF re-applier + the webhook. Recommend tighten/loosen of the route-policy gate. Output the setup+grader edit. - **Role 4 Governance Specialist** — the VS metadata gate + + Service + JWKS/authz/cronjob/canary/webhook/bleater-env. Tighten/loosen for thoroughness variance. Confirm baseline documents every key/path. - **Role 5 Coupling Reviewer (istio-critical)** — does the edit move ONLY the shared drift surface? If yes and the goal is decorrelation → REJECT; point at the existing `/etc/bleater` gate to tighten/loosen instead. If it adds/removes independent routing-or-governance difficulty via the gates → approve. - **Role 6 QC Reviewer** — expect 18 substantive pass + `v1_feedback` noise FAIL. Reject: naming drift/keys/filenames/route-policy specifics in task.yaml; any traffic check; requiring an exact drift/webhook/ConfigMap name; stacking a brand-new hidden lever. - **Role 7 Final Recommender** — produce the 8-point block (root cause / why avg shifted / smallest safe fix / expected rollout impact + correlation / expected QC impact / risk / exact files / exact lines). Then hand to `istio-iteration-loop` → `istio-task-guardian`. ## Operating procedure 1. Confirm trigger (Oracle failure → `istio-validation-debugger`). 2. Read grader/setup/task.yaml + rollout JSONs + ≥2 transcripts. 3. Diagnose (Role 1); look up the symptom. 4. Pick the lever — prefer tightening/loosening an existing `/etc/bleater`/metadata gate over a shared-drift lever; never stack a new mechanic. 5. Refine (Role 3/4). 6. Review (Role 5 Coupling + Role 6 QC). If rejected, re-pick. 7. Recommend (Role 7). 8. NEVER apply from this skill — hand to `istio-iteration-loop` → `istio-task-guardian`. ## Hard constraints Same as `istio-task-guardian`: no `ENABLE_ISTIO_BLEATER`; keep `ALLOWED_NAMESPACES="kube-system,bleater-env"`; keep the CRD install/establish block; keep the rbac COPY path + the `ubuntu-user-mesh-admission` ClusterRole; keep the webhook `failurePolicy: Ignore`; keep `/etc/bleater/*` 644 + `/opt/grader-*` 600; never name drift/faults/filenames/keys/route-policy specifics in task.yaml/baseline VALUES; **never add a behavioral check; never stack a new hidden lever**; air-gapped images only; `wait_consistent` ceilings `passes=5, gap=8s` (routing 220 / governance 200); don't simplify solution.sh. ## Preferred outcomes Healthy: 0.0/0.5/1.0 all present; avg –0.55; both subscores vary AND **decorrelated** (0.5 rollouts exist). Unhealthy: deadweight; avg > ; avg < ; perfect correlation (no 0.5); a structural impossibility (CRDs not establishing, a namespace unreachable, the webhook undeletable); any static-copy or traffic check creeping in; a new lever stacked instead of an existing one tuned. On an Unhealthy pattern two cycles running, halt and summarize.