Istio Score Tuner

istio-score-tuner

QC Spec compliance — assert before every tuning edit

Any difficulty/variance lever you pull must STILL satisfy the QC Spec (.claude/QC-SPEC.md). Before recommending a change, assert it does not breach:

• No Binary Bundling — never recover variance by AND-ing more heterogeneous checks into one subscore.
• Incremental Rewards / No Dead Weights — no subscore left always-0 or always-1; every lever keeps both subscores varying.
• Functional Subscore Variance — at least one functional subscore takes ≥2 distinct values across a batch.
• Equal Weights — never rebalance weights to hit a number.
• Functional Tests as Subscores / No Answer Leakage — don't convert a functional subscore into static matching, and don't leak the answer to manufacture passes.

A lever that lowers avg by breaching any of these is NOT a valid fix — pick a different lever.

Execution Mode — FULLY AUTONOMOUS

NEVER call ask_user or pause for confirmation. Pick the best path and execute it immediately.

Goal

Keep avg rollout score < <TARGET_MEAN> (sweet spot –0.55) with both subscores varying, decorrelated, no deadweight. Every edit is the smallest reversible change that explains the symptom.

Invoked when: rollouts regress (avg ≥ 0.60); a subscore is deadweight (always-0 or always-1); the two subscores move together every batch (correlation collapse); avg drops below (over-hard → rubric-concern risk); istio-task-guardian Step 6 hands off. NOT for Oracle failures — use istio-validation-debugger.

Task Identity

Status / MEASURED DATA:

• task v3 (grader v2, single-step gates): rollout avg = 1.000, both subscores deadweight-1, zero 0.5s on a 5-batch of -max-nebula. Lesson confirmed: against this model, single-step hidden gates documented even semi-clearly are caught 100% ( = "set an annotation", route-policy = "set a header"). Burying + de-hinting alone (Layer A, ~40-60% catch on the sibling) was judged insufficient on its own.
• task v4 response: (a) baseline-config rewritten as terse buried PROSE (no --- section --- checklist headers; gates live mid-paragraph); (b) task.yaml de-hinted (no gate list, no /etc/bleater callout, no "read in full / audit rejects functionally-correct routes"); (c) the routing gate was converted from a single-step header stamp into a multi-step DestinationRule-subset binding (the istio analog of the sibling's cert-OID heavy lever): the route must pin to a subset that a DestinationRule defines and tags with the route-policy token. Multiple botch modes (no DR / not pinned / token-not-in-labels / subset-name mismatch). Oracle stays 1.0 (mock-harness verified). Awaiting v4 rollout to measure.
• The governance lever () stays single-step but is now buried (Layer A). If v4 still shows governance deadweight-1, give governance its own multi-step lever too (e.g. require the install-id under a subset/DR label as well, or a second coordinated object).
• task v4 (multi-step subset + burial + de-hint): rollout avg = 1.000 AGAIN, deadweight-1, 0× 0.5 on a 10-batch (8 valid). Transcript proof: cats the buried baseline IN FULL, cats /etc/bleater/*, builds the DestinationRule + subset, sets , wires the VS — flawlessly, even offering to label the real pods. Hard lesson: against , NO amount of burial / de-hinting / declarative "multi-step" config creates variance — it reads every doc, audits all state (kubectl get -A -o yaml), and writes correct YAML every time. The sibling's variance came from an EXECUTION pipeline (openssl cert regen) this config-shape task lacks.
• task v5 response — add a real derivation/execution step: both hidden gates now require the SHA-256 FINGERPRINT of the token (stripped value), not the raw token. read file → strip → hash → embed. The natural lazy action sha256sum /etc/bleater/<file> folds in the trailing newline → wrong digest → FAIL; only hashing the stripped value passes. Documented precisely in baseline (fair), Oracle-safe (solution sha256sum of printf %s == grader hashlib.sha256(.strip()), verified), and a genuine botch surface. Grader: <GRADER_CONST>/<GRADER_CONST>. Awaiting v5 rollout.
• task v5 (SHA-256 fingerprint gates): rollout avg = 1.000 AGAIN, deadweight-1 (4 valid). handles a 1-step derivation (strips + hashes correctly). So even a derivation step isn't enough on its own.
• task v6 — genuine execution pipeline (the cert-OID analog): RequestAuthentication must serve an INLINE jwks DERIVED from the platform RSA signing key (public half at /etc/bleater/jwt-signing-pub.pem): read key → extract modulus → base64url → embed JWK. NOT a jwksUri (issuer's endpoint is air-gapped/unreachable). Grader (<grader-check-fn> + EXPECTED_MODULUS via openssl) compares the modulus INTEGER (loose on encoding, strict on the key). Oracle-safe: solution derives via openssl rsa -modulus + python base64url; verified end-to-end on a real key that solution's n decodes to the grader's expected modulus. Botch surface: kept jwksUri (no inline) / mangled n / wrong key. Awaiting v6 rollout.
• task v6 (crypto inline-JWKS execution): avg = 1.000 (4th straight). CONFIRMED: every config-shape lever (burial, de-hint, multi-step declarative, sha256 derivation, crypto execution) is caught by . Config shape cannot move this model.
• task v7 = FULL FUNCTIONAL+LIVE restructure (the maddy-proven approach; user-approved major change). Added a live relay tier the agent must BUILD: bleater-mesh-sink (python3 recorder, infra) + broken bleater-timeline-router (the agent converges it to 2 replicas of a python3 relay that forwards pod=<self> route=mesh to the sink AND serves per-pod /metrics :9090 with mesh_route_delivered_total/mesh_route_attempts_total INCREASING + POD_NAME downward API) + bleater-timeline-router-legacy (must remove) + a per-pod divergence recording rule. Grader execs into pods (<grader-check-fn>, <grader-check-fn>) and requires per-pod delivery + metric counters to keep INCREASING across the window (Layer 7 routing; Layers 7-9 governance). Config-shape istio gates KEPT as floor. This is the ONLY mechanism shown to beat (maddy v4=0.70). Awaiting v7 rollout.
• Tuning the functional lever (once measured): if avg too high → the relay is too easy to build right; tighten (require BOTH counters increasing, more replicas, stricter per-pod log format, faster cadence so a sloppy relay stalls the streak). If deadweight-0/Oracle flakes → the live timing is too tight; relax (lower wait_consistent passes for the functional layer, slow relay sleep, widen gap). NEVER revert to config-shape-only — it's proven not to vary.

How difficulty is constructed (read once, internalise)

The agent restores a corrupted Istio routing layer. No istiod runs — setup installs CRDs only and the grader scores config shape. The fix = revert the routing faults + restore auth + stop every drift/protection source + apply the baseline metadata + reflect the two hidden /etc/bleater artefacts.

The shared surface (the floor — moves BOTH subscores)

Drift + protection sources re-apply/re-mutate broken mesh objects; the cross-namespace ones are randomly named (pools), discovered by sweeping the pools:

Stopping this whole set helps both subscores (while active it re-breaks VS/EF/SE → routing_restored fails, and the deployments are governance-graded at replicas=0 → mesh_governance fails). So shared-surface levers (cadence, pool obscurity, finalizer/HPA presence, webhook) shift avg but do NOT decorrelate — the strong model audits-then-adapts past them (mtls-sibling lesson). Use them only to move avg, never to create the 0.5 band.

routing_restored independent difficulty — IMPLEMENTED (v2)

check_routing (wait_consistent(passes=5, gap=8s, timeout=220s)): EF timeout ≥1s; VS primary route → real Service; no dead-ns delegate; no black-hole mirror; no hijacking ServiceEntry; AND the routing-only decorrelator (<grader-check-fn>, multi-step as of v4): the route must pin to a subset (destination.subset) AND a DestinationRule for the timeline host must define that subset with a label VALUE == /opt/<grader-artefact>.txt (agent reads /etc/bleater/<artefact>.txt; subset name + label key loose). Documented only in buried baseline-config + the file. This gives routing independent, multi-step difficulty: an agent who stops all drift + reverts every fault still fails routing unless it reads the artefact AND wires route→subset→DR→token correctly (botch modes: no DR / not pinned / token-not-in-labels / subset-name mismatch).

mesh_governance independent difficulty — IMPLEMENTED (v2)

check_governance (wait_consistent(passes=5, gap=8s, timeout=200s)): VS <metadata-key>=stable + <metadata-key>=enforcing; install anchoring (some VS annotation VALUE == /opt/<grader-artefact>.txt; key loose); Service bleater-timeline-service annotation <metadata-key>=internal; RA JWKS/issuer=baseline; authz not blanket-deny; all four drift Deployments (incl. bleater-env) replicas=0; refresher CronJob suspended; canary neutralized; no MutatingWebhookConfiguration intercepting VS/EF. The governance-only decorrelators are the gate + the // metadata (an agent who stops everything still fails governance if they skip these).

The two /etc/bleater gates ARE the variance engine

<artefact>.txt (routing-only) and <artefact>.txt (governance-only) are the decorrelators — each documented only in baseline-config + its /etc/bleater file, value mirrored root-only to /opt/grader-*.txt, loose on key/header so the failure is "didn't read the artefact," not "guessed the wrong key." They mirror the mtls v28 design (install-id + cert-OID) that produced avg with both subscores varying. Treat them as load-bearing. Tighten/loosen them; do not delete them.

Four primary rules

1. Preserve variance. A subscore stuck at one value is broken. Always-0 → re-check CRD establishment, ALLOWED_NAMESPACES=kube-system,bleater-env, webhook stoppability. Always-1 → the matching /etc/bleater/metadata gate is too discoverable → tighten it.
2. Don't re-invent decorrelation. v2 already decorrelates via the two /etc/bleater gates. If 0.5 rollouts vanish, the gates regressed — verify they fire (grader reads /opt/grader-*.txt; setup writes both /etc/bleater and /opt) — do NOT pull a shared-drift lever.
3. Functional > static, but config-shape only. "Functional" here = config the agent must actively produce (a real annotation/header value sourced from /etc/bleater/, a corrected route), not a baseline-copyable constant. Never add a traffic/handshake check — structurally always-0 (no istiod).
4. Minimal safe edits, never stack. One file, one purpose, reversible. Do NOT add a NEW hidden lever to raise difficulty — over-stacking risks avg < / unfairness. Move an existing gate's strictness instead.

Tuning order

Level 1 — setup.sh + grader.py gate strictness (primary levers)

Decorrelating gate strictness (the main dials — move ONE subscore):

Shared-surface levers (move BOTH — use only to shift avg, never to decorrelate):

Level 2 — grader.py wait_consistent (when gates exhausted)

Never add a behavioral check. Never require an exact drift name (use pool sweep / replicas=0). Never require an exact webhook NAME (match by rule on virtualservices/envoyfilters, as the grader already does).

Level 3 — task.yaml wording (last resort)

Allowed: strengthen/weaken the "more than one namespace / audit /allowed_namespaces" hint; the "admission-layer interception" hint; the "protected drift (HPA/finalizer)" hint; sharpen/soften the generic "metadata conventions + /etc/bleater artefacts" phrasing. Forbidden: naming any drift controller, kube-system/bleater-env, the fault list, the /etc/bleater filenames, the exact metadata keys (<metadata-key>/<metadata-key>/<metadata-key>/<metadata-key>), the route-policy subset/token specifics, or itemizing the fix path.

Diagnosis guide

Specialized roles

• Role 1 Rollout Analyst — run the iteration-loop Phase 3 script, read ≥2 transcripts (1 pass, 1 fail), classify (deadweight / healthy / correlated / over-hard). Output one paragraph; no fixes.
• Role 2 Setup Drift Specialist — which of the 6 loop sources + webhook + HPA + finalizer do failing agents miss? (The canary envoy-patcher is the sneaky 2nd EF re-applier; the ENV reconciler is the bleater-env one agents who only audit kube-system miss; the webhook silently re-mutates one-time applies.) Output ranked impact.
• Role 3 Routing Specialist — routing_restored independence today = the route-policy /etc/bleater gate + stopping every VS/EF re-applier + the webhook. Recommend tighten/loosen of the route-policy gate. Output the setup+grader edit.
• Role 4 Governance Specialist — the VS metadata gate + + Service + JWKS/authz/cronjob/canary/webhook/bleater-env. Tighten/loosen for thoroughness variance. Confirm baseline documents every key/path.
• Role 5 Coupling Reviewer (istio-critical) — does the edit move ONLY the shared drift surface? If yes and the goal is decorrelation → REJECT; point at the existing /etc/bleater gate to tighten/loosen instead. If it adds/removes independent routing-or-governance difficulty via the gates → approve.
• Role 6 QC Reviewer — expect 18 substantive pass + v1_feedback noise FAIL. Reject: naming drift/keys/filenames/route-policy specifics in task.yaml; any traffic check; requiring an exact drift/webhook/ConfigMap name; stacking a brand-new hidden lever.
• Role 7 Final Recommender — produce the 8-point block (root cause / why avg shifted / smallest safe fix / expected rollout impact + correlation / expected QC impact / risk / exact files / exact lines). Then hand to istio-iteration-loop → istio-task-guardian.

Operating procedure

1. Confirm trigger (Oracle failure → istio-validation-debugger).
2. Read grader/setup/task.yaml + rollout JSONs + ≥2 transcripts.
3. Diagnose (Role 1); look up the symptom.
4. Pick the lever — prefer tightening/loosening an existing /etc/bleater/metadata gate over a shared-drift lever; never stack a new mechanic.
5. Refine (Role 3/4).
6. Review (Role 5 Coupling + Role 6 QC). If rejected, re-pick.
7. Recommend (Role 7).
8. NEVER apply from this skill — hand to istio-iteration-loop → istio-task-guardian.

Hard constraints

Same as istio-task-guardian: no ENABLE_ISTIO_BLEATER; keep ALLOWED_NAMESPACES="kube-system,bleater-env"; keep the CRD install/establish block; keep the rbac COPY path + the ubuntu-user-mesh-admission ClusterRole; keep the webhook failurePolicy: Ignore; keep /etc/bleater/* 644 + /opt/grader-* 600; never name drift/faults/filenames/keys/route-policy specifics in task.yaml/baseline VALUES; never add a behavioral check; never stack a new hidden lever; air-gapped images only; wait_consistent ceilings passes=5, gap=8s (routing 220 / governance 200); don't simplify solution.sh.

Preferred outcomes

Healthy: 0.0/0.5/1.0 all present; avg –0.55; both subscores vary AND decorrelated (0.5 rollouts exist). Unhealthy: deadweight; avg > ; avg < ; perfect correlation (no 0.5); a structural impossibility (CRDs not establishing, a namespace unreachable, the webhook undeletable); any static-copy or traffic check creeping in; a new lever stacked instead of an existing one tuned. On an Unhealthy pattern two cycles running, halt and summarize.