# Mtls Score Tuner

> Diagnose rollout regressions and tune difficulty for the variation-fanout-pipeline-break task (<TASK_ID>). Specializes in keeping avg rollout score < <TARGET_MEAN> with both subscores varying, avoiding deadweight, and not introducing hidden coupling between mtls_handshake and trust_governance. Picks the smallest safe lever for the observed symptom.

- Skill: `omonuj/mtls-score-tuner` (Agent Skill)
- Install (CLI): `npx skillmds@latest add omonuj/mtls-score-tuner`
- Raw SKILL.md: https://api.skillmd.com/api/skills/omonuj/mtls-score-tuner/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: omonuj (https://skillmd.com/u/omonuj)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/omonuj/mtls-score-tuner

---


# mtls-score-tuner

## QC Spec compliance — assert before every tuning edit

Any difficulty/variance lever you pull must STILL satisfy the QC Spec (`.claude/QC-SPEC.md`). Before recommending a change, assert it does not breach:
- **No Binary Bundling** — never recover variance by AND-ing more heterogeneous checks into one subscore (the v29→v30 trap).
- **Incremental Rewards / No Dead Weights** — no subscore left always-0 or always-1; every lever keeps both subscores varying.
- **Functional Subscore Variance** — at least one functional subscore takes ≥2 distinct values across a batch.
- **Equal Weights** — never rebalance weights to hit a number.
- **Functional Tests as Subscores / No Answer Leakage** — don't convert a functional subscore into static matching, and don't leak the answer to manufacture passes.

A lever that lowers avg by breaching any of these is NOT a valid fix — pick a different lever.

## Execution Mode — FULLY AUTONOMOUS

**NEVER call `ask_user` or pause for confirmation.** Pick the best path and execute it immediately.

## Goal

Tune task difficulty and variance safely. **The objective is not to "pass" — it is to keep avg rollout score < <TARGET_MEAN> with both subscores varying, neither deadweight, and no hidden coupling between them.** Every edit this skill proposes must be the smallest reversible change that explains the observed symptom.

This skill is invoked when:
- Rollouts regress (avg ≥ 0.60)
- A subscore becomes deadweight (always 0 or always 1)
- One subscore changes while tuning the other (coupling regression)
- `mtls-task-guardian` Step 6 hands off

This skill is **not** invoked for Oracle failures — those are validation issues, use `mtls-validation-debugger`.

## Trigger

| Symptom | Use this skill? |
|---|---|
| Rollout avg ≥ 0.60 | Yes |
| `mtls_handshake` always 0 or always 1 | Yes |
| `trust_governance` always 0 or always 1 | Yes |
| Both subscores stuck at the same value | Yes — investigate each independently |
| Oracle returns < 1.0 | No — use `mtls-validation-debugger` |
| Quality check below 18/20 (but ≥ 15) on the four known noise checks | No — accepted, see `mtls-task-guardian` Step 2 |
| Setup.sh syntax error | No — fix directly via `mtls-task-guardian` Step 1 |

## Task Identity

| Field | Value |
|---|---|
| **Task UUID** | `<TASK_ID>` |
| **Task slug** | `variation-fanout-pipeline-break` |
| **Scoring** | `mtls_handshake × 0.5 + trust_governance × 0.5`, both binary |
| **Target** | rollout `avg < <TARGET_MEAN>, both subscores in `{0, 1}` somewhere in the batch |

---

## How difficulty is constructed (read once, internalise)

The agent's job at runtime is to restore TLS trust against a hostile cluster. Both subscores are scored against the live world after the agent's solution runs. The difficulty comes from the following structural facts — every tuning lever moves one of them:

### `mtls_handshake` difficulty axis

What the agent must accomplish (functional outcome, verified by grader.py `check_mtls_handshake`):
1. Regenerate a CA with **Subject CN = `bleater-ca`** (the documented baseline identity) and a leaf cert whose SAN includes **`bleater-profile-service-mtls.bleater.svc.cluster.local`**
2. Patch Secret `bleater-profile-mtls` (keys `tls.crt`, `tls.key`) with the new material AND patch ConfigMap `bleater-trust-bundle` (key `ca.crt`) with the new CA
3. Bounce both `bleater-profile-service` AND `bleater-bleat-service` pods (`kubectl delete pod -l app=...`) — the kubelet's ~60s Secret-projection delay otherwise shields the running pod from any Secret-only patch
4. Stop **every** Secret-patching drift source — otherwise the patch gets reverted mid-window
5. Hold stable for `wait_consistent(passes=5, gap=15s, timeout=240s)` — ~75s continuous verified handshake from the bleat-service mtls-client sidecar to the profile-service mTLS endpoint, with chain verification + hostname verification
6. **(v19+ Strategy E)** Survive the mid-window drift injection: after the streak first reaches 2 successful passes, the grader patches the Secret with the wrong cert ONCE. Agent must have deployed a self-healing reconciler (e.g. `bleater-trust-guardian`) that re-applies the baseline cert within ~30s so the streak can rebuild. This is the new primary handshake-variance axis as of v19 — agents who scorched-earth without leaving a defence cannot pass.

Where the hidden difficulty comes from:
- **`node-telemetry-batcher`** is a CronJob in `kube-system` that fires every 60s. The 75s `wait_consistent` window guarantees it runs at least once mid-window if not stopped. Agents who don't audit `kube-system` get their Secret flipped, handshake fails. (See grader.py line 252-255.)
- **Strategy A (`bleater-profile-cache-sync`)** is a Deployment in `bleater` running a Python+urllib loop that PATCHes the Secret every 45s with the wrong-CA cert (setup.sh lines 1253-1290). It does NOT bounce the pod itself — it bites handshake because the 45s cadence is faster than the agent's stable window: the agent patches the Secret with the good cert and bounces the pod, then within 45-105s Strategy A patches the Secret back and the kubelet eventually re-projects the bad cert into the running pod. Not named in `profile-mtls-baseline-config.txt`. Agents who only stop the obvious controllers fail handshake on this delayed mechanism.
- **`cluster-trust-syncer`** Deployment in `kube-system` re-strips the trust bundle on a fast cycle. Same kube-system discovery axis.

### `trust_governance` difficulty axis

What the agent must accomplish (functional outcome, verified by grader.py `check_trust_governance`):

1. **Cert rotator CronJob** — delete `bleater-profile-cert-rotator` OR `spec.suspend=true`
2. **Three drift Deployments** must all have `replicas=0` AND `readyReplicas=0`:
   - `cluster-trust-syncer` in `kube-system`
   - `bleater-mesh-validator` in `kube-system`
   - `bleater-cert-reaper` in `bleater` — **protected by HPA `bleater-cert-reaper-pinner`**, must delete HPA first
3. **`bleater-profile-canary` mutation capability** must be neutralized — *either* remove the `cert-watcher` sidecar from the canary Deployment *or* delete the `bleater-profile-canary-runtime` ServiceAccount/Role/RoleBinding. Removing either alone is sufficient (grader.py line 333: `if sidecar_present and (sa or role or rb)`).
4. Hold no-drift for `wait_consistent(passes=5, gap=8s, timeout=200s)` — ~40s continuous

Where the hidden difficulty comes from:
- **The primary variance axis is controller-discovery scope.** Agents who only audit `bleater` namespace miss `cluster-trust-syncer` and `bleater-mesh-validator` in `kube-system` — both keep restoring drift, governance fails. The task.yaml prompt hints "Some of the reapply pressure may originate from outside the `bleater` namespace" and the `Dockerfile` grants `ALLOWED_NAMESPACES="kube-system"` — agents who don't follow the hint pay for it.
- **The HPA ordering trap** kills agents who `kubectl scale deployment bleater-cert-reaper --replicas=0` before deleting `bleater-cert-reaper-pinner` — the HPA restores `replicas=1` within seconds. The `[11:04]` incident-notes line leaks this without naming the HPA.
- **The canary OR-rule** rewards agents who notice they have two paths to neutralize the same mutation capability — they don't need to do both.

**Items deployed but not directly governance-graded** (enforced indirectly via handshake stability — don't try to add them to `check_trust_governance` without first reading the v9 lesson):
- `bleater-profile-cache-sync` Deployment (Strategy A)
- `node-telemetry-batcher` CronJob (kube-system)
- The `bleater-cert-reaper-pinner` HPA itself (it's a *blocker* for `cert-reaper`, not its own grade target)

**Inert baseline misdirections** (named in `profile-mtls-baseline-config.txt`, do not affect score):
- `cluster-config-sync` (CronJob in `kube-system`) — flagged in baseline as "leave untouched unless explicitly implicated"
- `bleater-profile-metrics-cache` (in `bleater`) — flagged as "read-only, no cert access"

**These two axes are the entire design.** Every tuning lever either sharpens one of them or accidentally collapses one of them. The job of this skill is to know which.

---

## Four primary rules

### Rule 1: Preserve variance

A subscore stuck at a single value across all rollouts is **broken**, not "easy" or "hard":
- Always-0: structurally impossible — agents have no path, no training signal
- Always-1: trivially passing — dead reward dimension

Healthy distribution:
- avg between <FLOOR> and 0.55
- both subscores appear at both 0 and 1 in any 5-rollout batch
- no subscore deadweight

### Rule 2: Avoid cross-coupling

Before proposing any change, ask: *does this lever touch the other subscore?*

Shared surfaces that cause coupling:
- The cert-reaper CronJob affects **both** subscores (governance: must be stopped; handshake: re-patches cert)
- Any drift source that bounces the profile-service pod affects handshake timing
- `wait_consistent` parameters on either subscore affect the timing window the other has to recover

Reject any change that has plausible blast radius into the other subscore without a written analysis of why it won't.

### Rule 3: Prefer functional variance over static checks

Functional variance is durable. Static variance evaporates the moment an agent reads the baseline config.

| Good (functional) | Bad (static) — v6 lesson |
|---|---|
| TLS handshake succeeds against the live service | Cert CN equals `bleater-ca` |
| Drift controller does not re-patch within `wait_consistent` window | Cert validity is exactly 365 days |
| Pod survives `wait_consistent(passes=5)` | Annotation value equals a known string |
| HPA pinner deleted before cert-reaper scaled | Replica count equals a fixed number |

### Rule 4: Minimal safe edits

Prefer:
- one-file edits
- one-purpose edits
- small reversible changes

Reject:
- multi-system rewrites in one cycle
- simultaneous setup + grader + task.yaml changes
- architectural redesigns mid-iteration

Per cycle, the workflow is: **read transcripts → identify root cause → propose smallest viable fix → estimate score impact + coupling risk + QC impact → apply.**

---

## Tuning order — pull the safest lever first

When you have a clear root cause, walk this list top-to-bottom and use the first applicable lever:

### Level 1 — setup.sh levers (safest)

| Lever | Effect | Direction | Notes |
|---|---|---|---|
**Verify the current value in setup.sh before editing — do not trust this table for live values.** The line numbers below are anchors; always `grep` the named variable to confirm.

**Levers that move `mtls_handshake` difficulty** (drift-source cadences — faster = harder):

| Lever (variable / cron field) | File:line | Current | Direction to raise difficulty | Floor — don't go below |
|---|---|---|---|---|
| `bleater-profile-cache-sync` Python loop (Strategy A) | setup.sh:1290 | `time.sleep(45)` | Decrease to 30s | ~30s (k8s API rate-limits below) |
| `bleater-cert-reaper` Python loop | setup.sh:1020 | `time.sleep(7)` | Decrease toward 5s | 5s |
| `bleater-mesh-validator` Python loop | setup.sh:767 | `time.sleep(20)` | Decrease to 15s | 10s |
| `cluster-trust-syncer` Python loop | setup.sh:630 | `time.sleep(5)` | **Already at floor** | 5s |
| `bleater-profile-cert-rotator` CronJob schedule | setup.sh:486 | `* * * * *` | **At k8s CronJob minimum (1 min)** | 1 min |
| `node-telemetry-batcher` CronJob schedule | setup.sh:828 | `*/1 * * * *` | **At k8s CronJob minimum (1 min)** | 1 min |
| `cert-watcher` sidecar (in `bleater-profile-canary`) | setup.sh:1110+ | Python loop, see block | Tighten the patch interval | k8s API rate limit |

**Levers that move `trust_governance` difficulty** (discoverability — *content* edits in heredocs, not numeric):

| Lever | File:line | What to edit | Direction to raise difficulty | Legitimacy guard |
|---|---|---|---|---|
| Baseline-config misdirection density | setup.sh:375 (`<<'BASELINE'` heredoc) | Add more plausible-looking inert entries (currently `cluster-config-sync`, `bleater-profile-metrics-cache`) | More misdirection ⇒ agents waste discovery budget on red herrings | Misdirection ≠ leaking. Adding a *real* drift controller name here = REJECT (v9 lesson) |
| Strategy A hiddenness in baseline | setup.sh:375 (same heredoc) | Confirm `cache-sync` does NOT appear by name | Maintained hiddenness is *the* handshake-variance lever for cache-sync | If `cache-sync` ever appears here, REJECT and revert |
| HPA trap hint strength | setup.sh:409 (`<<'NOTES'` incident-notes heredoc), `[11:04]` line | Make the wording vaguer | Vaguer ⇒ fewer agents catch the HPA ordering trap | Cannot name `HPA` or `pinner` |
| Cross-namespace hint in incident notes | setup.sh:409 (same heredoc), `[11:14]` line | Make vaguer | Vaguer ⇒ fewer agents audit kube-system | Cannot name `kube-system` |
| Canary hint in incident notes | setup.sh:409 (same heredoc), `[11:21]` line | Make vaguer | Vaguer ⇒ fewer agents notice canary mutation capability | Cannot name `cert-watcher` or `canary-runtime` |

**Bridge note on coupling:** Drift-source loop intervals (top table) primarily move *handshake* — governance just checks `replicas=0`, so the loop's speed doesn't matter once the agent stops the controller. But anything touching `bleater-cert-reaper` or `bleater-profile-canary` *also* affects governance grading because those two are governance-graded entities — run Coupling Reviewer (Role 5) before changing either.

### Level 2 — grader.py levers (use only when setup.sh is exhausted)

| Lever | Effect | Direction | Ceiling — do not exceed |
|---|---|---|---|
| `mtls_handshake` `wait_consistent passes` | More passes ⇒ narrower stable window | Current: `passes=5` | **`passes=5`** |
| `mtls_handshake` `wait_consistent gap` | Larger gap ⇒ must stay clean longer | Current: `gap=15` | **`gap=15s`** |
| `mtls_handshake` `wait_consistent timeout` | Lower timeout ⇒ less recovery budget | Current: `timeout=240` | **`timeout=240s`** (floor 180s; below this legitimate solutions flake) |
| `trust_governance` `wait_consistent passes` | More passes ⇒ controllers must stay stopped longer | Current: `passes=5` | **`passes=5`** |
| `trust_governance` `wait_consistent gap` | Larger gap ⇒ broader stability requirement | Current: `gap=8` | **`gap=8s`** |
| `trust_governance` `wait_consistent timeout` | Lower timeout ⇒ tighter window | Current: `timeout=200` | **`timeout=200s`** (floor 150s) |

**Always read the current values from `grader.py` (`check_mtls_handshake` last line, `check_trust_governance` last line) before editing.** The ceilings above are the empirical maxima from production runs; the floors are where flake starts to drive dead-zero.

Above any ceiling, legitimate solutions flake into dead-zero — variance collapses. The ceilings are empirical, not arbitrary.

### Level 3 — task.yaml wording (last resort, surgical only)

Touching `task.yaml` is extremely high-leverage — v9 proved that 5–10 words can swing variance by 60+ percentage points.

Allowed wording moves:
- Weaken or strengthen the hint about "paths that are not the most obvious to audit" (the kube-system discovery hint)
- Change a soft adjective (e.g. "all" → "every", or removing an emphasis)

Forbidden wording moves:
- Adding any drift-controller name (`cert-reaper`, `cache-sync`, `canary`, `hpa-pinner`)
- Adding the word `kube-system`
- Adding any RBAC-cleanup step
- Listing the fix path step-by-step

If you change `task.yaml`, change exactly one phrase and revert immediately if variance moves the wrong way.

---

## Diagnosis guide — symptom → root cause → fix

Always **read at least 2 rollout transcripts** before tuning. Score numbers alone don't reveal *why* agents succeeded or failed — transcript paths do. Look for the phrase the agent used right before failure (e.g. "I've verified the cert is stable" or "I checked the bleater namespace").

| Observed symptom | Most likely root cause | Correct lever |
|---|---|---|
| `mtls_handshake` always 1.0 | Strategy A re-patch interval too slow, OR agents reliably find `bleater-profile-cache-sync` | Speed up cache-sync loop in setup.sh; consider stronger baseline misdirection |
| `mtls_handshake` always 0.0 | Cert break is too deep, OR solution.sh has a regression | **Verify Oracle passes first**. If Oracle is 1.0, cert break is fine — investigate why agents can't recover. If Oracle ≠ 1.0, fix solution.sh via `mtls-validation-debugger` |
| `mtls_handshake` always 0.0 even with Oracle 1.0 | Drift loop interval is faster than grader's `wait_consistent` window for handshake | Slow the cache-sync loop in setup.sh, OR raise `gap` in grader (within ceiling) |
| `trust_governance` always 1.0 | Agents systematically find all controllers — most likely they're auditing `.allowed_namespaces` correctly | **Accepted if avg < <TARGET_MEAN>.** If not, consider renaming a controller to a less obvious name, or moving one to an unexpected namespace |
| `trust_governance` always 0.0 | Structural fault: `ALLOWED_NAMESPACES` missing from Dockerfile, OR kube-system RBAC absent | This is a `mtls-task-guardian` Step 0 issue — fix Dockerfile and re-verify kube-system Role in setup.sh |
| avg exactly 0.50 | One subscore deadweight-1, other deadweight-0 | Both are broken — diagnose each independently with the rows above |
| avg between 0.60 and 0.70 | Slightly too easy — usually `trust_governance` discoverability creeping up | Tighten `gap` for governance (within ceiling) before touching setup.sh |
| avg above 0.80 | One subscore probably deadweight-1 — run the variance check first | Use the analysis script from `mtls-iteration-loop` to confirm which |

---

## Hard constraints (NEVER violate — same as task-guardian)

The Why column explains the incident behind each rule:

| Constraint | Why |
|---|---|
| Do NOT add `ENABLE_ISTIO_BLEATER=true` | k3s startup failure → `score: 0, feedback: null` (setup crash) |
| Do NOT remove `ALLOWED_NAMESPACES="kube-system"` | `trust_governance` becomes structurally always-0 |
| Do NOT remove `COPY data/ubuntu-user-rbac.yaml` | Setup aborts before fault injection (DO NOT CHANGE block fails) |
| Do NOT name drift controllers in `task.yaml` or baseline config | v9 collapsed variance to always-1 with one explicit phrase |
| Do NOT add cert-content checks that only require *preserving* a baseline-documented value | v6 (fanout task) lesson: baseline documents → agents copy → dead-weight. This rule applies **only** when the agent's job is to preserve. In the mtls task agents must *regenerate* the CA (Subject CN), so the existing `CN=bleater-ca` and SAN checks are signal, not dead-weight. The grader docstring (line 132-134) calls this out explicitly. |
| Do NOT use external registry images for any drift source | Cluster is air-gapped — failed pull turns the drift source into a ghost |
| Do NOT scale `bleater-cert-reaper` before deleting `bleater-cert-reaper-pinner` HPA | HPA immediately restores `replicas=1` within seconds |
| Do NOT exceed `wait_consistent(passes=5, gap=15s, timeout=240s)` for handshake | Calibrated against `node-telemetry-batcher`'s 60s CronJob cadence. Tighter → legitimate solutions flake. |
| Do NOT exceed `wait_consistent(passes=5, gap=8s, timeout=200s)` for governance | Controllers can briefly flake on restart; above ceiling the score is noise |
| Do NOT simplify solution.sh to make Oracle easier | solution.sh is the oracle standard, not a tuning surface |

---

## What is NOT graded (do not add these to grader)

Adding any of these would re-introduce a v6-style dead-weight failure mode (the agent has nothing to actively produce — they just copy or leave the value alone):

- Cert validity period
- Replica count on any *non-drift* Deployment
- Annotation values on `bleater-profile-service` (Strategy C strips these intentionally — re-adding them as a check would create a copy-from-baseline trap)
- ConfigMap contents directly *as the gate* (the trust-bundle CA is checked as a regenerated artifact, not as a config-match)
- HPA spec contents (the HPA must be deleted, not modified — checking spec creates wrong incentives)

**What IS legitimately graded as cert content** (currently in `check_mtls_handshake`):
- Trust bundle CA Subject CN must equal `bleater-ca` — *agents regenerate the CA, so this catches regeneration with a wrong CN*
- Leaf cert SAN must include `bleater-profile-service-mtls.bleater.svc.cluster.local` — verified via `openssl s_client -verify_hostname`, *catches regeneration with a wrong SAN*
- Cert chain must verify against `/etc/bleater/trust.pem` — *catches a wrong-CA cert or stripped trust bundle*

The rule: cert-content checks are legitimate when the agent has to actively *produce* the value correctly. They become dead-weight when the agent's job is to *preserve* a value the baseline already documents.

---

## Specialized diagnostic roles

When the symptom is ambiguous, narrow the search by adopting one of these focused stances. Each role has a single question and a single artifact-of-interest — answer the question, write the artifact, then return.

### Role 1: Rollout Analyst

**Question:** What did agents actually do, and why did they fail or succeed?

**Inputs:** Latest 5+ rollout JSONs and transcripts for the current version.

**Process:**
1. Run the analysis script (from `mtls-iteration-loop` Phase 3) for the statistical summary
2. Read at least 2 transcripts — **including at least one passing one and one failing one** — to identify the dominant agent path
3. Classify the distribution: deadweight / healthy variance / unstable variance

**Output:** One paragraph: "Across N rollouts, avg=X, handshake pass rate Y%, governance Z%. Agents who passed did A. Agents who failed missed B." Do not propose fixes yet — only diagnose.

### Role 2: Setup Drift Specialist

**Question:** Of the drift sources setup.sh deploys, which are actually biting agents?

**Inputs:** setup.sh, the failing transcripts identified by the Rollout Analyst.

**Process:**
- For each drift source (`bleater-cert-reaper`, `bleater-profile-cache-sync`, kube-system Python loop, `profile-canary`, HPA pinner), check whether the failing transcripts mention discovering or stopping it
- The strongest variance driver is the one **most failed agents missed and most passed agents stopped**

**Output:** Ranked list of drift sources by variance-driving impact. Prefer setup.sh edits over grader edits.

### Role 3: Handshake Specialist

**Question:** What controls `mtls_handshake` variance, independent of governance?

**Process:**
- Two drift sources bite handshake within the 75s `wait_consistent` window, both by *timing*:
  1. **`node-telemetry-batcher`** CronJob in `kube-system` (60s cadence — Secret rotator; guaranteed to fire at least once mid-window)
  2. **Strategy A `bleater-profile-cache-sync`** Deployment (45s Python+urllib PATCH loop; faster than the agent's bounce-and-stabilise window)
- The v5 lesson said "Secret-only drift is shielded by kubelet projection delay (~60s)". That's true for a *one-shot* Secret-only patch on a freshly-bounced pod. But on a *recurring* cadence shorter than the wait_consistent window, the kubelet eventually re-projects the bad cert into the running pod. So the agent's options are: stop the source, or out-bounce its cadence (impractical at 45s).
- Inspect: each source's interval, the pod-delete trigger in cache-sync, grader `wait_consistent` window vs re-patch speed
- Inspect cert-regeneration correctness in solution.sh: SAN must include the headless FQDN; CA Subject CN must equal `bleater-ca`
- Reject: any *new* cert-content check that doesn't require active production (validity period, replicas, annotation values) — those are dead-weight

**Output:** Specific edit to one of: cache-sync timing, cache-sync pod-bounce mechanic, telemetry-batcher cadence, or solution.sh cert regeneration steps. Verify it doesn't widen governance discoverability.

### Role 4: Governance Specialist

**Question:** What controls `trust_governance` variance, independent of handshake?

**Process:**
- The five governance-graded entities are: `bleater-profile-cert-rotator` CronJob, `cluster-trust-syncer` Deployment (kube-system), `bleater-mesh-validator` Deployment (kube-system), `bleater-cert-reaper` Deployment (bleater, HPA-protected), and `bleater-profile-canary` mutation capability (sidecar OR `-runtime` RBAC).
- **Primary variance axis: kube-system discovery scope** — agents who only audit `bleater` miss `cluster-trust-syncer` + `bleater-mesh-validator`. The task.yaml hint ("may originate from outside the bleater namespace") and `ALLOWED_NAMESPACES="kube-system"` in the Dockerfile are the breadcrumbs.
- **Secondary axis: HPA ordering trap** — `bleater-cert-reaper-pinner` HPA must be deleted before scaling `bleater-cert-reaper` to 0. The `[11:04]` incident-notes line leaks the symptom without naming the HPA.
- **Tertiary axis: canary OR-rule** — sidecar removal *or* RBAC removal is sufficient, not both. Don't change the grader logic without a coupling review.
- Prefer live no-re-patch behavior (`wait_consistent` enforcement) over enumerating individual RBAC bindings (agents who deleted the canary Deployment outright will also pass).

**Output:** Either a setup.sh change to discoverability/hiddenness of one of the five graded entities, or a grader `wait_consistent` change within ceiling. Verify it doesn't shorten the handshake recovery window.

### Role 5: Coupling Reviewer

**Question:** What other subscore could the proposed edit accidentally move?

**Inputs:** The proposed edit and the current variance state of both subscores.

**Process:** Walk the shared-surface inventory and rule each one in or out:

- **The cert-reaper CronJob** touches both subscores. Any change to its cadence or naming affects handshake (re-patches the cert) *and* governance (must be stopped).
- **The cache-sync Deployment (Strategy A)** is a governance-discoverable controller AND the only drift source that bounces the profile-service pod. Changing its loop interval moves handshake variance; changing its discoverability moves governance variance; changing the pod-bounce step touches both.
- **The HPA pinner** is governance-only on the surface, but its deletion-ordering trap affects whether cert-reaper actually stops in time — and a still-running cert-reaper bleeds into handshake recovery.
- **`wait_consistent` parameters** on either subscore set the timing window the *other* subscore's drift has to respect. Tightening governance's window can make a slow-to-stabilise handshake fail.
- **Strategy A hiddenness in baseline config** — making it easier to find moves *both*: handshake gets easier (fewer agents get re-patched) and governance gets easier (fewer agents miss the controller).

**Output:** Either "no coupling risk" with the ruled-out shared surfaces listed, or "REJECT — proposed edit moves X via Y; propose alternative Z instead."

**Hard rule:** If you cannot confidently rule out coupling, reject the proposal. Coupling regressions are how v9 lost a week.

### Role 6: QC Reviewer

**Question:** Would this edit break the local quality check or create a new permanent-fail mode?

**Inputs:** The proposed edit + the current `horizon check-quality` baseline.

**Process:**

- Expected quality score: **18/20** with the four known permanent fails (`v1_feedback`, `behavior_in_task_documentation`, `behavior_in_tests`, sometimes `task_clarity`). All four trace to the LLM reviewer being unable to read the heredoc baseline config in setup.sh — unfixable without exposing drift-controller names, which kills variance.
- Predict the reviewer's reaction to this specific edit. The reviewer will flag and reject:
  - Any drift controller named in `task.yaml` or in the visible portions of setup.sh (v9 collapse)
  - Any new cert-content check in grader.py (v6 dead-weight)
  - Any RBAC-cleanup step itemised in the prompt (v9 collapse)
- A QC drop from 18 → 17 or 18 → 16 is acceptable if the edit fixes a real variance problem and no new permanent fail is introduced.
- A QC drop to 13 or below is unacceptable — investigate what broke before pushing.

**Output:** Either "QC delta: 0 (or +1)" with the predicted check that changes, or "REJECT — predicted QC drop to X because reviewer will flag Y."

### Role 7: Final Recommender

**Question:** What is the single safest change to apply now?

**Inputs:** The diagnoses from all preceding roles (Rollout Analyst + Setup Drift / Handshake / Governance Specialist + Coupling Reviewer + QC Reviewer).

**Output format** — produce exactly this 8-point block, nothing else:

1. **Root cause** — what specifically caused the observed symptom (one sentence)
2. **Why current avg shifted** — the score math (e.g. "handshake pass rate rose from 60% to 90% because cache-sync loop interval was raised from 45s → 60s in v14")
3. **Smallest safe fix** — exactly one edit, one file
4. **Expected rollout impact** — projected avg + projected per-subscore distribution
5. **Expected QC impact** — delta on the 18/20 baseline
6. **Risk assessment** — what could go wrong, ranked
7. **Exact files to modify** — file path(s), nothing else
8. **Exact lines/components affected** — line numbers or named blocks

Only after the Recommender's block is produced does the orchestrator (`mtls-iteration-loop`) hand off to `mtls-task-guardian` for the actual edit + push.

---

## Operating procedure — diagnose, propose, approve, apply

This is the diagnose-then-prescribe discipline. **NEVER edit immediately on a fresh diagnosis.** The whole point of this skill is to slow down between "I see a symptom" and "I apply a fix" — because every fast fix in the v1–v16 history was followed by a regression.

When invoked, walk these steps in order:

1. **Confirm the trigger** — Use the trigger table at the top. If the symptom is an Oracle failure, stop and hand off to `mtls-validation-debugger`. This skill is for rollout regressions only.
2. **Read** — grader.py, setup.sh, task.yaml, the latest rollout JSONs, **at least 2 rollout transcripts (1 pass, 1 fail)**. Score numbers alone never reveal why agents succeeded or failed.
3. **Diagnose** — Run Role 1 (Rollout Analyst) for the distribution summary. Look up the symptom in the diagnosis guide. Identify whether the issue is grader strictness, setup drift intensity, hidden coupling, weak functional signal, over-bundling, or deadweight variance.
4. **Pick the lever** — Walk the Tuning order top-to-bottom. Use Level 1 (setup.sh) unless exhausted; Level 2 (grader.py) only if necessary; Level 3 (`task.yaml`) only as last resort.
5. **Refine** — Adopt the matching specialist role (Setup Drift / Handshake / Governance) to pin down the specific edit.
6. **Review for coupling and QC risk** — Run the proposed edit through Role 5 (Coupling Reviewer) and Role 6 (QC Reviewer). If either rejects, return to step 4 and pick a different lever.
7. **Recommend** — Role 7 (Final Recommender) produces the 8-point block.
8. **NEVER apply the edit from this skill.** Hand the Recommender's block to `mtls-iteration-loop`, which routes it through `mtls-task-guardian` for syntax check, push, and validation. The discipline is **diagnose → propose → approve → apply**, with the approval gate being explicit — either the user, or (in an autonomous loop) verification that all preceding role-outputs concur and none rejected.

The cost of this discipline is one extra minute per cycle. The benefit is not re-making v9 — which cost a week and required a regression rollback.

---

## Verification cheat sheet

| Hypothesis | Probe |
|---|---|
| "Subscore X is deadweight" | `python -c "import json,glob; from collections import defaultdict; v=defaultdict(set); [v[k].add(s) for f in glob.glob('tasks/.../.rollouts/vN/*.json') for k,s in json.loads(json.load(open(f))['grade_result'])['subscores'].items()]; print({k: sorted(s) for k,s in v.items()})"` — any subscore with `len==1` is deadweight |
| "Cache-sync is the variance driver for handshake" | Read 3 failing handshake transcripts — at least 2 must mention failing to stop cache-sync |
| "kube-system Python loop is the variance driver for governance" | Read 3 failing governance transcripts — most must have audited only `bleater` |
| "The Strategy A misdirection still works" | grep the baseline config heredoc in setup.sh — `cache-sync` must not appear by name |
| "Wait_consistent hasn't drifted past ceiling" | `grep -E "wait_consistent" grader.py` — handshake: `passes=5, gap=15, timeout=240`; governance: `passes=5, gap=8, timeout=200`. Different values = ceiling drift. |
| "No *new* preserve-baseline cert check has crept into grader" | `grep -iE "NotAfter\|NotBefore\|Issuer\|Serial" grader.py` — must return nothing. (`Subject` / `CN=bleater-ca` / SAN are expected — they catch regeneration mistakes.) |
| "Setup events are wiped" | `grep -n "kubectl delete events" setup.sh` — must appear near the end, before the hide-grader block |

---

## Preferred outcomes

**Healthy:**

- 0.0 / 0.5 / 1.0 all represented across a 5-rollout batch
- avg <FLOOR>–0.55 (with `< <TARGET_MEAN>` as the firm ceiling)
- Both `mtls_handshake` and `trust_governance` vary (both 0 and 1 appear)
- Failures are *legitimate* — transcripts show agents missing a discoverable drift source, not hitting a structural impossibility
- Reviewer complaints stay within the four known permanent fails

**Unhealthy:**

- Any subscore deadweight-0 (structural impossibility — re-check Dockerfile invariants)
- Any subscore deadweight-1 (trivially passing — controller too discoverable, or `wait_consistent` window too forgiving)
- avg > <TARGET_MEAN> (Nebula creator workflow rejection)
- Static-only grading (cert CN/SAN/validity checks creeping back in — v6 regression)
- Excessive binary bundling (a subscore is the AND of too many independent gates)
- Hidden undiscoverable requirements (e.g. a drift controller in a namespace not in `ALLOWED_NAMESPACES`)
- Coupling drift (changes to handshake levers moving governance, or vice versa — v9 regression)

If you observe any Unhealthy pattern, **halt the loop and write a summary** instead of pushing another fix. Two cycles in a row chasing an Unhealthy pattern is a sign the lever choice was wrong, not that you need another lever.

---

## Important philosophy

**Do NOT optimise for:**

- *Perfect grading* — the task is graded by behaviour, not perfection
- *Maximal strictness* — tighter is easy; tighter is also brittle and breeds dead-zero
- *Maximal realism* — more controllers, more loops, more knobs — each one is a coupling channel waiting to fire
- *Passing a single rollout batch* — variance across batches is the signal; one batch is noise

**DO optimise for:**

- *Useful training signal* — both subscores vary, both directions are reachable
- *Controllable variance* — when you turn a lever, you can predict which subscore moves and by how much
- *Fairness* — every failing agent had a discoverable path to success they didn't take, not an impossible obstacle
- *Solvability* — solution.sh scores 1.0 reliably; the task is hard, not impossible
- *Reviewer survivability* — the four known permanent fails are accepted; no new permanent fails introduced
- *Stable rollout distributions* — the same version, run twice a week apart, produces the same shape

The tension between these two lists is the entire job of this skill. Every Hard constraint above is a scar from a time someone optimised for the first list and broke the second.