# Maddy Score Tuner

> Diagnose rollout regressions and tune difficulty for the maddy-delivery-split-brain task (<TASK_ID>). Specializes in keeping avg rollout score < <TARGET_MEAN> with both subscores varying, avoiding deadweight, and not letting the shared drift-controller surface collapse delivery_consistency and observability_governance into one correlated axis. Picks the smallest safe lever for the observed symptom.

- Skill: `omonuj/maddy-score-tuner` (Agent Skill)
- Install (CLI): `npx skillmds@latest add omonuj/maddy-score-tuner`
- Raw SKILL.md: https://api.skillmd.com/api/skills/omonuj/maddy-score-tuner/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: omonuj (https://skillmd.com/u/omonuj)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/omonuj/maddy-score-tuner

---


# maddy-score-tuner

## QC Spec compliance — assert before every tuning edit

Any difficulty/variance lever you pull must STILL satisfy the QC Spec (`.claude/QC-SPEC.md`). Before recommending a change, assert it does not breach:
- **No Binary Bundling** — never recover variance by AND-ing more heterogeneous checks into one subscore.
- **Incremental Rewards / No Dead Weights** — no subscore left always-0 or always-1; every lever keeps both subscores varying.
- **Functional Subscore Variance** — at least one functional subscore takes ≥2 distinct values across a batch.
- **Equal Weights** — never rebalance weights to hit a number.
- **Functional Tests as Subscores / No Answer Leakage** — don't convert a functional subscore into static matching, and don't leak the answer to manufacture passes.

A lever that lowers avg by breaching any of these is NOT a valid fix — pick a different lever.

## Execution Mode — FULLY AUTONOMOUS

**NEVER call `ask_user` or pause for confirmation.** Pick the best path and execute it immediately.

## Goal

Tune task difficulty and variance safely. **The objective is not to "pass" — it is to keep avg rollout score < <TARGET_MEAN> with both subscores varying, neither deadweight, and the two subscores decorrelated.** Every edit must be the smallest reversible change that explains the observed symptom.

Invoked when:
- Rollouts regress (avg ≥ 0.60)
- A subscore becomes deadweight (always 0 or always 1)
- The two subscores move together every batch (correlation collapse — the maddy-specific failure mode)
- `maddy-task-guardian` Step 6 hands off

NOT invoked for Oracle failures — use `maddy-validation-debugger`.

## Task Identity

| Field | Value |
|---|---|
| **Task UUID** | `<TASK_ID>` |
| **Task slug** | `maddy-delivery-split-brain` |
| **Scoring** | `delivery_consistency × 0.5 + observability_governance × 0.5`, both binary |
| **Target** | rollout `avg < <TARGET_MEAN>, both subscores in `{0,1}` somewhere in the batch, and NOT perfectly correlated |

---

## How difficulty is constructed (read once, internalise)

The agent's runtime job: diagnose a Maddy SMTP split-brain (one replica on smarthost config, one stale replica on direct-delivery config behind the same Service), converge it, make it observable, and stop the automation that keeps re-creating the stale replica. Both subscores are graded against the live world after the agent's solution runs.

### The shared surface (the thing that makes this task tricky to tune)

Three drift controllers re-apply a byte-identical `maddy-legacy` Deployment on a loop. Names are randomized (pool stem + numeric suffix); discover by behaviour:

| Drift source | Namespace | Cadence | Discover by | Scenario key |
|---|---|---|---|---|
| kube-system reconciler | `kube-system` | 5s | annotation `bleater.io/purpose: "maddy rollout convergence"` | `reconciler` |
| bleater-env reconciler | `bleater-env` | 20s | annotation `bleater.io/purpose: "maddy rollout convergence"` | `env_reconciler` |
| canary `config-pinner` sidecar | `bleater` | 11s | container named `config-pinner`; RBAC `<canary>-runtime` | `canary` / `canary_runtime` |

**These three are graded by BOTH subscores.** While active they re-spawn the legacy replica → `delivery_consistency` Layer 1/2 fails. They are also directly checked at `replicas=0` by `observability_governance` Layers 4/5/6. So **stopping the drift helps both subscores simultaneously.** If drift-discovery becomes the *only* hard thing, the two subscores correlate: an agent who finds all three passes both, one who misses one fails both → avg swings to 0 or 1 as a block, and you lose independent signal.

### `delivery_consistency` difficulty axis

`check_delivery_consistency` (`wait_consistent(passes=5, gap=8s, timeout=220s)`):
1. `maddy-config` (`maddy.conf`) is the smarthost config (`relay_mode = smarthost`, not `direct`)
2. NO `app=maddy` pod mounts `maddy-config-legacy`
3. `maddy-legacy` Deployment absent / `replicas=0`
4. `maddy` Deployment converged: `spec.replicas >= 2`, `updated == ready == spec`, `observedGeneration` current
5. ≥ 2 Ready `app=maddy` pods mount `maddy-config`

**Independent (decorrelating) lever:** the rollout-convergence requirement (scale `maddy` to 2 + all pods Ready on smarthost). An agent can find and stop all drift yet still fail consistency if they delete `maddy-legacy` but never scale `maddy` from 1 → 2, or leave the rollout half-applied. This is the part of `delivery_consistency` that does NOT depend on drift discovery.

### `observability_governance` difficulty axis

`check_observability_governance` (`wait_consistent(passes=5, gap=8s, timeout=200s)`):
1. `maddy` container exposes pod identity (`POD_NAME` via `fieldRef metadata.name`)
2. `maddy-gitops-source` `values.yaml` = `replicas: 2` + `smarthost`, not `mode: direct`
3. a per-pod divergence rule ConfigMap exists — real PromQL (`- record:`/`- alert:`/`expr:`) + signal (`rate`/`success`/`delivery`/`divergence`) + per-pod dimension (`by (pod)` etc.)
4. kube-system reconciler `replicas=0`
5. canary mutation path neutralized (sidecar OR `-runtime` RBAC)
6. bleater-env reconciler `replicas=0`

**Independent (decorrelating) levers:** the three observability artifacts (Layers 1–3). These have NOTHING to do with drift — an agent who stops all drift still fails governance if they don't add `POD_NAME`, fix the GitOps CM, and write a valid per-pod PromQL rule. **The per-pod-rule gate (Layer 3) is the strongest governance-only variance driver** — many agents will write a vague ConfigMap with the word "pod" in a comment and fail the `by (pod)` requirement.

### Where the hidden difficulty comes from

- **Cross-namespace discovery** — agents who only audit `bleater` miss the `kube-system` AND `bleater-env` reconcilers. The bleater-env one is doubly hidden: different namespace + innocuous non-maddy name (only the `bleater.io/purpose` annotation links it). The task.yaml hint ("Reapply pressure may originate in more than one namespace — check all namespaces in `.allowed_namespaces`") and the incident notes are the breadcrumbs.
- **The per-pod PromQL rule** — the `by (pod)` + rule-key requirement filters agents who hand-wave the observability ask.
- **Rollout convergence** — agents who fix the config but forget to actually scale `maddy` to 2.

**These are the entire design.** Every lever sharpens or collapses one of them.

### ⚠️ Structural deadweight check (do this FIRST when governance is stuck at 0)

If `observability_governance` is **always 0**, suspect the `bleater-env` reachability bug before anything else: if `ALLOWED_NAMESPACES` doesn't grant the agent `bleater-env`, Layer 6 is unstoppable and the whole subscore is structurally impossible. This is a `maddy-task-guardian` Step 0 / Dockerfile issue, NOT a tuning lever. Verify on a live container (guardian Step 4b) before touching grader/setup.

---

## Four primary rules

### Rule 1: Preserve variance
A subscore stuck at one value is **broken**. Always-0 = structurally impossible (re-check bleater-env reachability + Dockerfile). Always-1 = trivially passing. Healthy: avg <FLOOR>–0.55; both subscores show 0 and 1 in any 5-rollout batch.

### Rule 2: Avoid correlation collapse (the maddy-specific rule)
Before any change ask: *does this lever touch ONLY the shared drift surface?* If yes, it moves both subscores together and worsens correlation. Prefer the **decorrelating levers**:
- to move `delivery_consistency` alone → the rollout-convergence requirement
- to move `observability_governance` alone → the three observability artifacts (esp. the per-pod-rule gate)

Reject a change whose only effect is "make drift easier/harder to find" unless the explicit goal is to move avg and you accept both subscores shifting.

### Rule 3: Prefer functional variance over static checks
Functional (does the converged state hold under live drift; does a real PromQL `by(pod)` rule exist) is durable. Static (an exact ConfigMap name, an exact label value the baseline documents) evaporates the moment an agent reads the baseline. The per-pod-rule check is name-agnostic on purpose — keep it that way.

### Rule 4: Minimal safe edits
One file, one purpose, small, reversible. Reject multi-system rewrites, simultaneous setup+grader+task.yaml changes, architectural redesigns mid-iteration.

---

## Tuning order — pull the safest lever first

### Level 1 — setup.sh levers (safest)

**Verify the current value in setup.sh before editing — grep the named variable.**

**Levers that move the SHARED drift axis (move BOTH subscores — use to shift avg, not to decorrelate):**

| Lever | Where | Current | Raise difficulty | Floor |
|---|---|---|---|---|
| kube-system reconciler cadence | `emit_drift_deploy "${RECONCILER}" kube-system ... 5` | `5s` | already fast | 5s (API rate limits) |
| bleater-env reconciler cadence | `emit_drift_deploy "${ENV_RECONCILER}" bleater-env ... 20` | `20s` | decrease toward 10s | 10s |
| canary config-pinner cadence | sidecar loop `sleep 11` | `11s` | decrease toward 8s | 8s |
| bleater-env name-pool innocuousness | `ENV_RECONCILER_STEMS` | innocuous non-maddy stems | keep innocuous (vaguer = harder to find) | must stay annotation-discoverable |
| baseline-config misdirection density | `<<BASELINE` heredoc | — | add plausible inert entries | misdirection ≠ leaking a real drift name |

**Levers that move `observability_governance` ALONE (decorrelating — preferred for governance):**

| Lever | Where | Direction | Guard |
|---|---|---|---|
| GitOps source staleness obviousness | `maddy-gitops-source` heredoc (`replicas: 1`, `mode: direct`) | keep the stale values; this is the artifact the agent must fix | don't pre-fill the correct values |
| Per-pod-rule expectation strength | task.yaml Expected Outcome wording about "per-pod divergence signal" | vaguer ⇒ fewer agents produce a valid `by(pod)` rule | already in Expected Outcome — do not name a ConfigMap |
| `POD_NAME` hint strength | incident notes "logs ... no pod name" line | vaguer ⇒ fewer agents add the downward-API env | cannot name `fieldRef` / `downward API` |

**Levers that move `delivery_consistency` ALONE (decorrelating — preferred for consistency):**

| Lever | Where | Direction | Guard |
|---|---|---|---|
| Initial `maddy` replicas | `maddy` Deployment `replicas: 1` | keep at 1 so the agent must actively scale to 2 | this is the convergence lever; don't pre-set to 2 |
| Rollout-readiness pressure | `maddy` container resources / readiness | minor | don't make pods un-schedulable |

### Level 2 — grader.py levers (use only when setup.sh is exhausted)

| Lever | Effect | Current | Ceiling |
|---|---|---|---|
| `delivery_consistency` `wait_consistent gap` | larger ⇒ must stay clean longer | `gap=8` | **`gap=8s`** |
| `delivery_consistency` `wait_consistent timeout` | lower ⇒ less recovery budget | `timeout=220` | floor 180s |
| `observability_governance` `wait_consistent` | same | `passes=5, gap=8, timeout=200` | floor 150s |
| Per-pod-rule strictness (Layer 3) | the `by (pod)` + rule-key + signal triple-gate | current triple | **governance-only lever** — loosening raises governance pass rate, tightening lowers it. Do NOT require an exact ConfigMap name. |
| `INTENDED_REPLICAS` | bar for convergence | `2` | leave at 2 unless task.yaml changes too |

Above any ceiling, legitimate solutions flake into dead-zero (drift restart churn). The ceilings are empirical.

### Level 3 — task.yaml wording (last resort, surgical only)

Highest-leverage — a few words swing variance a lot.

Allowed: strengthen/weaken the "more than one namespace" hint; soften/sharpen the "per-pod divergence signal" or "pod identity in logs" phrasing.

Forbidden: naming any drift controller or the `bleater.io/purpose` value; naming `bleater-env` or `kube-system` explicitly as where drift lives; naming `config-pinner`; itemizing the fix path step-by-step; naming `fieldRef`/`POD_NAME`/an exact ConfigMap name.

Change exactly one phrase; revert immediately if variance moves the wrong way.

---

## Diagnosis guide — symptom → root cause → fix

Always **read at least 2 rollout transcripts** (1 pass, 1 fail) before tuning.

| Observed symptom | Most likely root cause | Correct lever |
|---|---|---|
| `delivery_consistency` always 1 | Drift too easy to find AND agents reliably scale to 2 | Speed up the bleater-env/canary cadence (shared) OR make the bleater-env name vaguer; if convergence is the gap, it's already minimal |
| `delivery_consistency` always 0 (Oracle passes) | A drift source re-spawns the legacy pod faster than agents stop it, OR a source is undiscoverable | Confirm all three are annotation/sidecar-discoverable; if a cadence is below floor, raise it slightly |
| `observability_governance` always 0 (Oracle passes) | **bleater-env unreachable (structural)** OR per-pod-rule gate too strict for legitimate agents | FIRST verify bleater-env reachability (guardian Step 0). Only then consider loosening Layer 3 |
| `observability_governance` always 1 | Per-pod rule + discovery too easy | Tighten the per-pod-rule requirement is risky (already strict); prefer making bleater-env discovery harder |
| Both subscores move together every batch (avg ~0 or ~1, never 0.5) | **Correlation collapse** — drift discovery is the only hard thing | Add independent difficulty: keep `maddy` initial replicas=1 (consistency-only) and ensure the per-pod-rule gate bites (governance-only). Do NOT pull a shared-drift lever |
| avg 0.60–0.70 | Slightly too easy | Tighten a governance-only or consistency-only lever before a shared one |
| avg > <TARGET_MEAN> | A subscore is probably deadweight-1 | Run the variance script first to confirm which |

---

## Specialized diagnostic roles

### Role 1: Rollout Analyst
**Q:** What did agents actually do, and why pass/fail? Run the analysis script (from `maddy-iteration-loop` Phase 3), read ≥2 transcripts (≥1 pass, ≥1 fail), classify the distribution (deadweight / healthy / correlated). Output one paragraph; no fixes yet.

### Role 2: Setup Drift Specialist
**Q:** Of the three drift sources, which are actually biting? For each (kube-system reconciler, bleater-env reconciler, canary config-pinner) check whether failing transcripts mention discovering/stopping it. The bleater-env one is usually the most-missed (innocuous name + namespace). Output: ranked variance impact. Prefer setup.sh edits.

### Role 3: Consistency Specialist
**Q:** What controls `delivery_consistency` variance independent of governance? Inspect: do failing agents stop all drift but forget to scale `maddy` to 2? Is the legacy replica re-spawning inside the window? The decorrelating lever is the convergence requirement (initial replicas=1). Output: a setup.sh/grader edit that moves consistency without touching governance discoverability.

### Role 4: Governance Specialist
**Q:** What controls `observability_governance` variance independent of consistency? The three artifacts: `POD_NAME` env, `maddy-gitops-source` edit, per-pod PromQL rule. Read failing transcripts: did agents write a ConfigMap with "pod" in a comment but no `by (pod)` expr (Layer-3 miss)? Did they skip the downward API? **First rule out bleater-env reachability as a structural deadweight.** Output: a task.yaml hint tweak or a Layer-3 strictness change (within reason).

### Role 5: Coupling Reviewer (the maddy-critical role)
**Q:** Does the proposed edit move ONLY the shared drift surface? Walk the inventory:
- Drift cadence/discoverability → moves BOTH subscores (correlation risk). Only acceptable when the explicit goal is shifting avg.
- Initial `maddy` replicas → consistency-only (safe to decorrelate)
- The three observability artifacts → governance-only (safe to decorrelate)
- `wait_consistent` on one subscore → affects only that subscore's window (drift restart churn can flake both if both windows shrink)
Output: "no correlation risk — moves only X" OR "REJECT — this is a shared-surface lever; it will move both subscores; propose a decorrelating alternative."

### Role 6: QC Reviewer
**Q:** Would this edit break local quality or add a permanent fail? Expected: 18 substantive checks pass; the `v1_feedback` section FAILs as accepted noise (heredoc-invisible baseline config + annotation-based discovery). Reject: naming a drift controller / `bleater.io/purpose` / `bleater-env` / `config-pinner` in `task.yaml` or visible setup; requiring an exact ConfigMap name in the per-pod-rule check. Output: "QC delta 0/+1" or "REJECT — reviewer will flag Y."

### Role 7: Final Recommender
Produce exactly this 8-point block:
1. **Root cause** (one sentence)
2. **Why avg shifted** (the score math)
3. **Smallest safe fix** (one edit, one file)
4. **Expected rollout impact** (projected avg + per-subscore distribution + correlation)
5. **Expected QC impact** (delta on the 18-check baseline)
6. **Risk assessment** (ranked)
7. **Exact files to modify**
8. **Exact lines/components affected**

Only after the Recommender's block does the orchestrator (`maddy-iteration-loop`) hand off to `maddy-task-guardian` for the edit + push.

---

## Operating procedure — diagnose, propose, approve, apply

**NEVER edit immediately on a fresh diagnosis.** Walk in order:
1. **Confirm trigger** — Oracle failure → hand to `maddy-validation-debugger`. This skill is rollout regressions only.
2. **Read** — grader.py, setup.sh, task.yaml, latest rollout JSONs, ≥2 transcripts (1 pass, 1 fail).
3. **Diagnose** — Role 1 for the distribution; look up the symptom; decide if it's deadweight, correlation collapse, grader strictness, or drift intensity.
4. **Pick the lever** — Level 1 (setup.sh) unless exhausted; Level 2 (grader) only if necessary; Level 3 (task.yaml) last. Prefer a decorrelating lever unless intentionally shifting avg.
5. **Refine** — adopt the matching specialist role (Consistency / Governance / Setup Drift).
6. **Review** — Role 5 (Coupling) and Role 6 (QC). If either rejects, return to step 4.
7. **Recommend** — Role 7's 8-point block.
8. **NEVER apply from this skill.** Hand to `maddy-iteration-loop` → `maddy-task-guardian` for syntax check, push, validation.

---

## Hard constraints (same as task-guardian)

| Constraint | Why |
|---|---|
| Do NOT add `ENABLE_ISTIO_BLEATER=true` | k3s startup crash → `score: 0, feedback: null` |
| Do NOT remove `kube-system` from `ALLOWED_NAMESPACES` | governance Layer 4 always-0 |
| Do NOT remove `bleater-env` reachability once added | governance Layer 6 unsolvable |
| Do NOT remove `COPY data/ubuntu-user-rbac.yaml` | setup aborts (DO NOT CHANGE block) |
| Do NOT name drift controllers / `bleater.io/purpose` / `config-pinner` / `bleater-env` in `task.yaml` or baseline heredoc | collapses `observability_governance` to always-1 |
| Do NOT require an exact ConfigMap name in the per-pod-rule check | static dead-weight; keep it name-agnostic with the `by(pod)` functional requirement |
| Do NOT use external registry images for drift | air-gapped — failed pull turns a drift source into a ghost |
| Do NOT exceed `wait_consistent(passes=5, gap=8s)` (consistency `timeout=220`, governance `timeout=200`) | above ceiling legitimate solutions flake to dead-zero |
| Do NOT simplify solution.sh to make Oracle pass | solution.sh is the oracle standard, not a tuning surface |

---

## What is NOT graded (do not add these to grader)

- Exact ConfigMap names (per-pod rule is name-agnostic by design)
- Replica count on the drift Deployments beyond `==0`
- The egress NetworkPolicy contents (scenario realism, not graded)
- `maddy.conf` fields beyond the `relay_mode = smarthost` / not-`direct` markers
- Annotation/label values that the baseline documents (copy-from-baseline dead-weight)

## Preferred outcomes

**Healthy:** 0.0 / 0.5 / 1.0 all represented across a 5-rollout batch; avg <FLOOR>–0.55; both subscores vary AND are not perfectly correlated; failures are legitimate (a discoverable drift source missed, or a vague per-pod rule, or forgot to scale to 2).

**Unhealthy:** any subscore deadweight; avg > <TARGET_MEAN>; the two subscores correlate perfectly every batch (correlation collapse); a structural impossibility (bleater-env unreachable); static-only grading creeping in.

If you observe an Unhealthy pattern, **halt and write a summary** rather than pushing another fix. Two cycles chasing the same Unhealthy pattern means the lever choice was wrong.