Mtls Score Tuner

mtls-score-tuner

QC Spec compliance — assert before every tuning edit

Any difficulty/variance lever you pull must STILL satisfy the QC Spec (.claude/QC-SPEC.md). Before recommending a change, assert it does not breach:

• No Binary Bundling — never recover variance by AND-ing more heterogeneous checks into one subscore (the v29→v30 trap).
• Incremental Rewards / No Dead Weights — no subscore left always-0 or always-1; every lever keeps both subscores varying.
• Functional Subscore Variance — at least one functional subscore takes ≥2 distinct values across a batch.
• Equal Weights — never rebalance weights to hit a number.
• Functional Tests as Subscores / No Answer Leakage — don't convert a functional subscore into static matching, and don't leak the answer to manufacture passes.

A lever that lowers avg by breaching any of these is NOT a valid fix — pick a different lever.

Execution Mode — FULLY AUTONOMOUS

NEVER call ask_user or pause for confirmation. Pick the best path and execute it immediately.

Goal

Tune task difficulty and variance safely. The objective is not to "pass" — it is to keep avg rollout score < with both subscores varying, neither deadweight, and no hidden coupling between them. Every edit this skill proposes must be the smallest reversible change that explains the observed symptom.

This skill is invoked when:

• Rollouts regress (avg ≥ 0.60)
• A subscore becomes deadweight (always 0 or always 1)
• One subscore changes while tuning the other (coupling regression)
• mtls-task-guardian Step 6 hands off

This skill is not invoked for Oracle failures — those are validation issues, use mtls-validation-debugger.

Trigger

Task Identity

How difficulty is constructed (read once, internalise)

The agent's job at runtime is to restore TLS trust against a hostile cluster. Both subscores are scored against the live world after the agent's solution runs. The difficulty comes from the following structural facts — every tuning lever moves one of them:

mtls_handshake difficulty axis

What the agent must accomplish (functional outcome, verified by grader.py check_mtls_handshake):

1. Regenerate a CA with Subject CN = bleater-ca (the documented baseline identity) and a leaf cert whose SAN includes bleater-profile-service-mtls.bleater.svc.cluster.local
2. Patch Secret bleater-profile-mtls (keys tls.crt, tls.key) with the new material AND patch ConfigMap bleater-trust-bundle (key ca.crt) with the new CA
3. Bounce both bleater-profile-service AND bleater-bleat-service pods (kubectl delete pod -l app=...) — the kubelet's ~60s Secret-projection delay otherwise shields the running pod from any Secret-only patch
4. Stop every Secret-patching drift source — otherwise the patch gets reverted mid-window
5. Hold stable for wait_consistent(passes=5, gap=15s, timeout=240s) — ~75s continuous verified handshake from the bleat-service mtls-client sidecar to the profile-service mTLS endpoint, with chain verification + hostname verification
6. (v19+ Strategy E) Survive the mid-window drift injection: after the streak first reaches 2 successful passes, the grader patches the Secret with the wrong cert ONCE. Agent must have deployed a self-healing reconciler (e.g. bleater-trust-guardian) that re-applies the baseline cert within ~30s so the streak can rebuild. This is the new primary handshake-variance axis as of v19 — agents who scorched-earth without leaving a defence cannot pass.

Where the hidden difficulty comes from:

• node-telemetry-batcher is a CronJob in kube-system that fires every 60s. The 75s wait_consistent window guarantees it runs at least once mid-window if not stopped. Agents who don't audit kube-system get their Secret flipped, handshake fails. (See grader.py line 252-255.)
• Strategy A (bleater-profile-cache-sync) is a Deployment in bleater running a Python+urllib loop that PATCHes the Secret every 45s with the wrong-CA cert (setup.sh lines 1253-1290). It does NOT bounce the pod itself — it bites handshake because the 45s cadence is faster than the agent's stable window: the agent patches the Secret with the good cert and bounces the pod, then within 45-105s Strategy A patches the Secret back and the kubelet eventually re-projects the bad cert into the running pod. Not named in profile-mtls-baseline-config.txt. Agents who only stop the obvious controllers fail handshake on this delayed mechanism.
• cluster-trust-syncer Deployment in kube-system re-strips the trust bundle on a fast cycle. Same kube-system discovery axis.

trust_governance difficulty axis

What the agent must accomplish (functional outcome, verified by grader.py check_trust_governance):

1. Cert rotator CronJob — delete bleater-profile-cert-rotator OR spec.suspend=true
2. Three drift Deployments must all have replicas=0 AND readyReplicas=0:
   • cluster-trust-syncer in kube-system
   • bleater-mesh-validator in kube-system
   • bleater-cert-reaper in bleater — protected by HPA bleater-cert-reaper-pinner, must delete HPA first
3. bleater-profile-canary mutation capability must be neutralized — either remove the cert-watcher sidecar from the canary Deployment or delete the bleater-profile-canary-runtime ServiceAccount/Role/RoleBinding. Removing either alone is sufficient (grader.py line 333: if sidecar_present and (sa or role or rb)).
4. Hold no-drift for wait_consistent(passes=5, gap=8s, timeout=200s) — ~40s continuous

Where the hidden difficulty comes from:

• The primary variance axis is controller-discovery scope. Agents who only audit bleater namespace miss cluster-trust-syncer and bleater-mesh-validator in kube-system — both keep restoring drift, governance fails. The task.yaml prompt hints "Some of the reapply pressure may originate from outside the bleater namespace" and the Dockerfile grants ALLOWED_NAMESPACES="kube-system" — agents who don't follow the hint pay for it.
• The HPA ordering trap kills agents who kubectl scale deployment bleater-cert-reaper --replicas=0 before deleting bleater-cert-reaper-pinner — the HPA restores replicas=1 within seconds. The [11:04] incident-notes line leaks this without naming the HPA.
• The canary OR-rule rewards agents who notice they have two paths to neutralize the same mutation capability — they don't need to do both.

Items deployed but not directly governance-graded (enforced indirectly via handshake stability — don't try to add them to check_trust_governance without first reading the v9 lesson):

• bleater-profile-cache-sync Deployment (Strategy A)
• node-telemetry-batcher CronJob (kube-system)
• The bleater-cert-reaper-pinner HPA itself (it's a blocker for cert-reaper, not its own grade target)

Inert baseline misdirections (named in profile-mtls-baseline-config.txt, do not affect score):

• cluster-config-sync (CronJob in kube-system) — flagged in baseline as "leave untouched unless explicitly implicated"
• bleater-profile-metrics-cache (in bleater) — flagged as "read-only, no cert access"

These two axes are the entire design. Every tuning lever either sharpens one of them or accidentally collapses one of them. The job of this skill is to know which.

Four primary rules

Rule 1: Preserve variance

A subscore stuck at a single value across all rollouts is broken, not "easy" or "hard":

• Always-0: structurally impossible — agents have no path, no training signal
• Always-1: trivially passing — dead reward dimension

Healthy distribution:

• avg between and 0.55
• both subscores appear at both 0 and 1 in any 5-rollout batch
• no subscore deadweight

Rule 2: Avoid cross-coupling

Before proposing any change, ask: does this lever touch the other subscore?

Shared surfaces that cause coupling:

• The cert-reaper CronJob affects both subscores (governance: must be stopped; handshake: re-patches cert)
• Any drift source that bounces the profile-service pod affects handshake timing
• wait_consistent parameters on either subscore affect the timing window the other has to recover

Reject any change that has plausible blast radius into the other subscore without a written analysis of why it won't.

Rule 3: Prefer functional variance over static checks

Functional variance is durable. Static variance evaporates the moment an agent reads the baseline config.

Rule 4: Minimal safe edits

Prefer:

• one-file edits
• one-purpose edits
• small reversible changes

Reject:

• multi-system rewrites in one cycle
• simultaneous setup + grader + task.yaml changes
• architectural redesigns mid-iteration

Per cycle, the workflow is: read transcripts → identify root cause → propose smallest viable fix → estimate score impact + coupling risk + QC impact → apply.

Tuning order — pull the safest lever first

When you have a clear root cause, walk this list top-to-bottom and use the first applicable lever:

Level 1 — setup.sh levers (safest)

Levers that move mtls_handshake difficulty (drift-source cadences — faster = harder):

Levers that move trust_governance difficulty (discoverability — content edits in heredocs, not numeric):

Bridge note on coupling: Drift-source loop intervals (top table) primarily move handshake — governance just checks replicas=0, so the loop's speed doesn't matter once the agent stops the controller. But anything touching bleater-cert-reaper or bleater-profile-canary also affects governance grading because those two are governance-graded entities — run Coupling Reviewer (Role 5) before changing either.

Level 2 — grader.py levers (use only when setup.sh is exhausted)

Always read the current values from grader.py (check_mtls_handshake last line, check_trust_governance last line) before editing. The ceilings above are the empirical maxima from production runs; the floors are where flake starts to drive dead-zero.

Above any ceiling, legitimate solutions flake into dead-zero — variance collapses. The ceilings are empirical, not arbitrary.

Level 3 — task.yaml wording (last resort, surgical only)

Touching task.yaml is extremely high-leverage — v9 proved that 5–10 words can swing variance by 60+ percentage points.

Allowed wording moves:

• Weaken or strengthen the hint about "paths that are not the most obvious to audit" (the kube-system discovery hint)
• Change a soft adjective (e.g. "all" → "every", or removing an emphasis)

Forbidden wording moves:

• Adding any drift-controller name (cert-reaper, cache-sync, canary, hpa-pinner)
• Adding the word kube-system
• Adding any RBAC-cleanup step
• Listing the fix path step-by-step

If you change task.yaml, change exactly one phrase and revert immediately if variance moves the wrong way.

Diagnosis guide — symptom → root cause → fix

Always read at least 2 rollout transcripts before tuning. Score numbers alone don't reveal why agents succeeded or failed — transcript paths do. Look for the phrase the agent used right before failure (e.g. "I've verified the cert is stable" or "I checked the bleater namespace").

Hard constraints (NEVER violate — same as task-guardian)

The Why column explains the incident behind each rule:

What is NOT graded (do not add these to grader)

Adding any of these would re-introduce a v6-style dead-weight failure mode (the agent has nothing to actively produce — they just copy or leave the value alone):

• Cert validity period
• Replica count on any non-drift Deployment
• Annotation values on bleater-profile-service (Strategy C strips these intentionally — re-adding them as a check would create a copy-from-baseline trap)
• ConfigMap contents directly as the gate (the trust-bundle CA is checked as a regenerated artifact, not as a config-match)
• HPA spec contents (the HPA must be deleted, not modified — checking spec creates wrong incentives)

What IS legitimately graded as cert content (currently in check_mtls_handshake):

• Trust bundle CA Subject CN must equal bleater-ca — agents regenerate the CA, so this catches regeneration with a wrong CN
• Leaf cert SAN must include bleater-profile-service-mtls.bleater.svc.cluster.local — verified via openssl s_client -verify_hostname, catches regeneration with a wrong SAN
• Cert chain must verify against /etc/bleater/trust.pem — catches a wrong-CA cert or stripped trust bundle

The rule: cert-content checks are legitimate when the agent has to actively produce the value correctly. They become dead-weight when the agent's job is to preserve a value the baseline already documents.

Specialized diagnostic roles

When the symptom is ambiguous, narrow the search by adopting one of these focused stances. Each role has a single question and a single artifact-of-interest — answer the question, write the artifact, then return.

Role 1: Rollout Analyst

Question: What did agents actually do, and why did they fail or succeed?

Inputs: Latest 5+ rollout JSONs and transcripts for the current version.

Process:

1. Run the analysis script (from mtls-iteration-loop Phase 3) for the statistical summary
2. Read at least 2 transcripts — including at least one passing one and one failing one — to identify the dominant agent path
3. Classify the distribution: deadweight / healthy variance / unstable variance

Output: One paragraph: "Across N rollouts, avg=X, handshake pass rate Y%, governance Z%. Agents who passed did A. Agents who failed missed B." Do not propose fixes yet — only diagnose.

Role 2: Setup Drift Specialist

Question: Of the drift sources setup.sh deploys, which are actually biting agents?

Inputs: setup.sh, the failing transcripts identified by the Rollout Analyst.

Process:

• For each drift source (bleater-cert-reaper, bleater-profile-cache-sync, kube-system Python loop, profile-canary, HPA pinner), check whether the failing transcripts mention discovering or stopping it
• The strongest variance driver is the one most failed agents missed and most passed agents stopped

Output: Ranked list of drift sources by variance-driving impact. Prefer setup.sh edits over grader edits.

Role 3: Handshake Specialist

Question: What controls mtls_handshake variance, independent of governance?

Process:

• Two drift sources bite handshake within the 75s wait_consistent window, both by timing:
  1. node-telemetry-batcher CronJob in kube-system (60s cadence — Secret rotator; guaranteed to fire at least once mid-window)
  2. Strategy A bleater-profile-cache-sync Deployment (45s Python+urllib PATCH loop; faster than the agent's bounce-and-stabilise window)
• The v5 lesson said "Secret-only drift is shielded by kubelet projection delay (~60s)". That's true for a one-shot Secret-only patch on a freshly-bounced pod. But on a recurring cadence shorter than the wait_consistent window, the kubelet eventually re-projects the bad cert into the running pod. So the agent's options are: stop the source, or out-bounce its cadence (impractical at 45s).
• Inspect: each source's interval, the pod-delete trigger in cache-sync, grader wait_consistent window vs re-patch speed
• Inspect cert-regeneration correctness in solution.sh: SAN must include the headless FQDN; CA Subject CN must equal bleater-ca
• Reject: any new cert-content check that doesn't require active production (validity period, replicas, annotation values) — those are dead-weight

Output: Specific edit to one of: cache-sync timing, cache-sync pod-bounce mechanic, telemetry-batcher cadence, or solution.sh cert regeneration steps. Verify it doesn't widen governance discoverability.

Role 4: Governance Specialist

Question: What controls trust_governance variance, independent of handshake?

Process:

• The five governance-graded entities are: bleater-profile-cert-rotator CronJob, cluster-trust-syncer Deployment (kube-system), bleater-mesh-validator Deployment (kube-system), bleater-cert-reaper Deployment (bleater, HPA-protected), and bleater-profile-canary mutation capability (sidecar OR -runtime RBAC).
• Primary variance axis: kube-system discovery scope — agents who only audit bleater miss cluster-trust-syncer + bleater-mesh-validator. The task.yaml hint ("may originate from outside the bleater namespace") and ALLOWED_NAMESPACES="kube-system" in the Dockerfile are the breadcrumbs.
• Secondary axis: HPA ordering trap — bleater-cert-reaper-pinner HPA must be deleted before scaling bleater-cert-reaper to 0. The [11:04] incident-notes line leaks the symptom without naming the HPA.
• Tertiary axis: canary OR-rule — sidecar removal or RBAC removal is sufficient, not both. Don't change the grader logic without a coupling review.
• Prefer live no-re-patch behavior (wait_consistent enforcement) over enumerating individual RBAC bindings (agents who deleted the canary Deployment outright will also pass).

Output: Either a setup.sh change to discoverability/hiddenness of one of the five graded entities, or a grader wait_consistent change within ceiling. Verify it doesn't shorten the handshake recovery window.

Role 5: Coupling Reviewer

Question: What other subscore could the proposed edit accidentally move?

Inputs: The proposed edit and the current variance state of both subscores.

Process: Walk the shared-surface inventory and rule each one in or out:

• The cert-reaper CronJob touches both subscores. Any change to its cadence or naming affects handshake (re-patches the cert) and governance (must be stopped).
• The cache-sync Deployment (Strategy A) is a governance-discoverable controller AND the only drift source that bounces the profile-service pod. Changing its loop interval moves handshake variance; changing its discoverability moves governance variance; changing the pod-bounce step touches both.
• The HPA pinner is governance-only on the surface, but its deletion-ordering trap affects whether cert-reaper actually stops in time — and a still-running cert-reaper bleeds into handshake recovery.
• wait_consistent parameters on either subscore set the timing window the other subscore's drift has to respect. Tightening governance's window can make a slow-to-stabilise handshake fail.
• Strategy A hiddenness in baseline config — making it easier to find moves both: handshake gets easier (fewer agents get re-patched) and governance gets easier (fewer agents miss the controller).

Output: Either "no coupling risk" with the ruled-out shared surfaces listed, or "REJECT — proposed edit moves X via Y; propose alternative Z instead."

Hard rule: If you cannot confidently rule out coupling, reject the proposal. Coupling regressions are how v9 lost a week.

Role 6: QC Reviewer

Question: Would this edit break the local quality check or create a new permanent-fail mode?

Inputs: The proposed edit + the current horizon check-quality baseline.

Process:

• Expected quality score: 18/20 with the four known permanent fails (v1_feedback, behavior_in_task_documentation, behavior_in_tests, sometimes task_clarity). All four trace to the LLM reviewer being unable to read the heredoc baseline config in setup.sh — unfixable without exposing drift-controller names, which kills variance.
• Predict the reviewer's reaction to this specific edit. The reviewer will flag and reject:
  • Any drift controller named in task.yaml or in the visible portions of setup.sh (v9 collapse)
  • Any new cert-content check in grader.py (v6 dead-weight)
  • Any RBAC-cleanup step itemised in the prompt (v9 collapse)
• A QC drop from 18 → 17 or 18 → 16 is acceptable if the edit fixes a real variance problem and no new permanent fail is introduced.
• A QC drop to 13 or below is unacceptable — investigate what broke before pushing.

Output: Either "QC delta: 0 (or +1)" with the predicted check that changes, or "REJECT — predicted QC drop to X because reviewer will flag Y."

Role 7: Final Recommender

Question: What is the single safest change to apply now?

Inputs: The diagnoses from all preceding roles (Rollout Analyst + Setup Drift / Handshake / Governance Specialist + Coupling Reviewer + QC Reviewer).

Output format — produce exactly this 8-point block, nothing else:

1. Root cause — what specifically caused the observed symptom (one sentence)
2. Why current avg shifted — the score math (e.g. "handshake pass rate rose from 60% to 90% because cache-sync loop interval was raised from 45s → 60s in v14")
3. Smallest safe fix — exactly one edit, one file
4. Expected rollout impact — projected avg + projected per-subscore distribution
5. Expected QC impact — delta on the 18/20 baseline
6. Risk assessment — what could go wrong, ranked
7. Exact files to modify — file path(s), nothing else
8. Exact lines/components affected — line numbers or named blocks

Only after the Recommender's block is produced does the orchestrator (mtls-iteration-loop) hand off to mtls-task-guardian for the actual edit + push.

Operating procedure — diagnose, propose, approve, apply

This is the diagnose-then-prescribe discipline. NEVER edit immediately on a fresh diagnosis. The whole point of this skill is to slow down between "I see a symptom" and "I apply a fix" — because every fast fix in the v1–v16 history was followed by a regression.

When invoked, walk these steps in order:

1. Confirm the trigger — Use the trigger table at the top. If the symptom is an Oracle failure, stop and hand off to mtls-validation-debugger. This skill is for rollout regressions only.
2. Read — grader.py, setup.sh, task.yaml, the latest rollout JSONs, at least 2 rollout transcripts (1 pass, 1 fail). Score numbers alone never reveal why agents succeeded or failed.
3. Diagnose — Run Role 1 (Rollout Analyst) for the distribution summary. Look up the symptom in the diagnosis guide. Identify whether the issue is grader strictness, setup drift intensity, hidden coupling, weak functional signal, over-bundling, or deadweight variance.
4. Pick the lever — Walk the Tuning order top-to-bottom. Use Level 1 (setup.sh) unless exhausted; Level 2 (grader.py) only if necessary; Level 3 (task.yaml) only as last resort.
5. Refine — Adopt the matching specialist role (Setup Drift / Handshake / Governance) to pin down the specific edit.
6. Review for coupling and QC risk — Run the proposed edit through Role 5 (Coupling Reviewer) and Role 6 (QC Reviewer). If either rejects, return to step 4 and pick a different lever.
7. Recommend — Role 7 (Final Recommender) produces the 8-point block.
8. NEVER apply the edit from this skill. Hand the Recommender's block to mtls-iteration-loop, which routes it through mtls-task-guardian for syntax check, push, and validation. The discipline is diagnose → propose → approve → apply, with the approval gate being explicit — either the user, or (in an autonomous loop) verification that all preceding role-outputs concur and none rejected.

The cost of this discipline is one extra minute per cycle. The benefit is not re-making v9 — which cost a week and required a regression rollback.

Verification cheat sheet

Preferred outcomes

Healthy:

• 0.0 / 0.5 / 1.0 all represented across a 5-rollout batch
• avg –0.55 (with < <TARGET_MEAN> as the firm ceiling)
• Both mtls_handshake and trust_governance vary (both 0 and 1 appear)
• Failures are legitimate — transcripts show agents missing a discoverable drift source, not hitting a structural impossibility
• Reviewer complaints stay within the four known permanent fails

Unhealthy:

• Any subscore deadweight-0 (structural impossibility — re-check Dockerfile invariants)
• Any subscore deadweight-1 (trivially passing — controller too discoverable, or wait_consistent window too forgiving)
• avg > (Nebula creator workflow rejection)
• Static-only grading (cert CN/SAN/validity checks creeping back in — v6 regression)
• Excessive binary bundling (a subscore is the AND of too many independent gates)
• Hidden undiscoverable requirements (e.g. a drift controller in a namespace not in ALLOWED_NAMESPACES)
• Coupling drift (changes to handshake levers moving governance, or vice versa — v9 regression)

If you observe any Unhealthy pattern, halt the loop and write a summary instead of pushing another fix. Two cycles in a row chasing an Unhealthy pattern is a sign the lever choice was wrong, not that you need another lever.

Important philosophy

Do NOT optimise for:

• Perfect grading — the task is graded by behaviour, not perfection
• Maximal strictness — tighter is easy; tighter is also brittle and breeds dead-zero
• Maximal realism — more controllers, more loops, more knobs — each one is a coupling channel waiting to fire
• Passing a single rollout batch — variance across batches is the signal; one batch is noise

DO optimise for:

• Useful training signal — both subscores vary, both directions are reachable
• Controllable variance — when you turn a lever, you can predict which subscore moves and by how much
• Fairness — every failing agent had a discoverable path to success they didn't take, not an impossible obstacle
• Solvability — solution.sh scores 1.0 reliably; the task is hard, not impossible
• Reviewer survivability — the four known permanent fails are accepted; no new permanent fails introduced
• Stable rollout distributions — the same version, run twice a week apart, produces the same shape

The tension between these two lists is the entire job of this skill. Every Hard constraint above is a scar from a time someone optimised for the first list and broke the second.