Maddy Score Tuner

maddy-score-tuner

QC Spec compliance — assert before every tuning edit

Any difficulty/variance lever you pull must STILL satisfy the QC Spec (.claude/QC-SPEC.md). Before recommending a change, assert it does not breach:

• No Binary Bundling — never recover variance by AND-ing more heterogeneous checks into one subscore.
• Incremental Rewards / No Dead Weights — no subscore left always-0 or always-1; every lever keeps both subscores varying.
• Functional Subscore Variance — at least one functional subscore takes ≥2 distinct values across a batch.
• Equal Weights — never rebalance weights to hit a number.
• Functional Tests as Subscores / No Answer Leakage — don't convert a functional subscore into static matching, and don't leak the answer to manufacture passes.

A lever that lowers avg by breaching any of these is NOT a valid fix — pick a different lever.

Execution Mode — FULLY AUTONOMOUS

NEVER call ask_user or pause for confirmation. Pick the best path and execute it immediately.

Goal

Tune task difficulty and variance safely. The objective is not to "pass" — it is to keep avg rollout score < with both subscores varying, neither deadweight, and the two subscores decorrelated. Every edit must be the smallest reversible change that explains the observed symptom.

Invoked when:

• Rollouts regress (avg ≥ 0.60)
• A subscore becomes deadweight (always 0 or always 1)
• The two subscores move together every batch (correlation collapse — the maddy-specific failure mode)
• maddy-task-guardian Step 6 hands off

NOT invoked for Oracle failures — use maddy-validation-debugger.

Task Identity

How difficulty is constructed (read once, internalise)

The agent's runtime job: diagnose a Maddy SMTP split-brain (one replica on smarthost config, one stale replica on direct-delivery config behind the same Service), converge it, make it observable, and stop the automation that keeps re-creating the stale replica. Both subscores are graded against the live world after the agent's solution runs.

The shared surface (the thing that makes this task tricky to tune)

Three drift controllers re-apply a byte-identical maddy-legacy Deployment on a loop. Names are randomized (pool stem + numeric suffix); discover by behaviour:

These three are graded by BOTH subscores. While active they re-spawn the legacy replica → delivery_consistency Layer 1/2 fails. They are also directly checked at replicas=0 by observability_governance Layers 4/5/6. So stopping the drift helps both subscores simultaneously. If drift-discovery becomes the only hard thing, the two subscores correlate: an agent who finds all three passes both, one who misses one fails both → avg swings to 0 or 1 as a block, and you lose independent signal.

delivery_consistency difficulty axis

check_delivery_consistency (wait_consistent(passes=5, gap=8s, timeout=220s)):

1. maddy-config (maddy.conf) is the smarthost config (relay_mode = smarthost, not direct)
2. NO app=maddy pod mounts maddy-config-legacy
3. maddy-legacy Deployment absent / replicas=0
4. maddy Deployment converged: spec.replicas >= 2, updated == ready == spec, observedGeneration current
5. ≥ 2 Ready app=maddy pods mount maddy-config

Independent (decorrelating) lever: the rollout-convergence requirement (scale maddy to 2 + all pods Ready on smarthost). An agent can find and stop all drift yet still fail consistency if they delete maddy-legacy but never scale maddy from 1 → 2, or leave the rollout half-applied. This is the part of delivery_consistency that does NOT depend on drift discovery.

observability_governance difficulty axis

check_observability_governance (wait_consistent(passes=5, gap=8s, timeout=200s)):

1. maddy container exposes pod identity (POD_NAME via fieldRef metadata.name)
2. maddy-gitops-source values.yaml = replicas: 2 + smarthost, not mode: direct
3. a per-pod divergence rule ConfigMap exists — real PromQL (- record:/- alert:/expr:) + signal (rate/success/delivery/divergence) + per-pod dimension (by (pod) etc.)
4. kube-system reconciler replicas=0
5. canary mutation path neutralized (sidecar OR -runtime RBAC)
6. bleater-env reconciler replicas=0

Independent (decorrelating) levers: the three observability artifacts (Layers 1–3). These have NOTHING to do with drift — an agent who stops all drift still fails governance if they don't add POD_NAME, fix the GitOps CM, and write a valid per-pod PromQL rule. The per-pod-rule gate (Layer 3) is the strongest governance-only variance driver — many agents will write a vague ConfigMap with the word "pod" in a comment and fail the by (pod) requirement.

Where the hidden difficulty comes from

• Cross-namespace discovery — agents who only audit bleater miss the kube-system AND bleater-env reconcilers. The bleater-env one is doubly hidden: different namespace + innocuous non-maddy name (only the bleater.io/purpose annotation links it). The task.yaml hint ("Reapply pressure may originate in more than one namespace — check all namespaces in .allowed_namespaces") and the incident notes are the breadcrumbs.
• The per-pod PromQL rule — the by (pod) + rule-key requirement filters agents who hand-wave the observability ask.
• Rollout convergence — agents who fix the config but forget to actually scale maddy to 2.

These are the entire design. Every lever sharpens or collapses one of them.

⚠️ Structural deadweight check (do this FIRST when governance is stuck at 0)

If observability_governance is always 0, suspect the bleater-env reachability bug before anything else: if ALLOWED_NAMESPACES doesn't grant the agent bleater-env, Layer 6 is unstoppable and the whole subscore is structurally impossible. This is a maddy-task-guardian Step 0 / Dockerfile issue, NOT a tuning lever. Verify on a live container (guardian Step 4b) before touching grader/setup.

Four primary rules

Rule 1: Preserve variance

A subscore stuck at one value is broken. Always-0 = structurally impossible (re-check bleater-env reachability + Dockerfile). Always-1 = trivially passing. Healthy: avg –0.55; both subscores show 0 and 1 in any 5-rollout batch.

Rule 2: Avoid correlation collapse (the maddy-specific rule)

Before any change ask: does this lever touch ONLY the shared drift surface? If yes, it moves both subscores together and worsens correlation. Prefer the decorrelating levers:

• to move delivery_consistency alone → the rollout-convergence requirement
• to move observability_governance alone → the three observability artifacts (esp. the per-pod-rule gate)

Reject a change whose only effect is "make drift easier/harder to find" unless the explicit goal is to move avg and you accept both subscores shifting.

Rule 3: Prefer functional variance over static checks

Functional (does the converged state hold under live drift; does a real PromQL by(pod) rule exist) is durable. Static (an exact ConfigMap name, an exact label value the baseline documents) evaporates the moment an agent reads the baseline. The per-pod-rule check is name-agnostic on purpose — keep it that way.

Rule 4: Minimal safe edits

One file, one purpose, small, reversible. Reject multi-system rewrites, simultaneous setup+grader+task.yaml changes, architectural redesigns mid-iteration.

Tuning order — pull the safest lever first

Level 1 — setup.sh levers (safest)

Verify the current value in setup.sh before editing — grep the named variable.

Levers that move the SHARED drift axis (move BOTH subscores — use to shift avg, not to decorrelate):

Levers that move observability_governance ALONE (decorrelating — preferred for governance):

Levers that move delivery_consistency ALONE (decorrelating — preferred for consistency):

Level 2 — grader.py levers (use only when setup.sh is exhausted)

Above any ceiling, legitimate solutions flake into dead-zero (drift restart churn). The ceilings are empirical.

Level 3 — task.yaml wording (last resort, surgical only)

Highest-leverage — a few words swing variance a lot.

Allowed: strengthen/weaken the "more than one namespace" hint; soften/sharpen the "per-pod divergence signal" or "pod identity in logs" phrasing.

Forbidden: naming any drift controller or the bleater.io/purpose value; naming bleater-env or kube-system explicitly as where drift lives; naming config-pinner; itemizing the fix path step-by-step; naming fieldRef/POD_NAME/an exact ConfigMap name.

Change exactly one phrase; revert immediately if variance moves the wrong way.

Diagnosis guide — symptom → root cause → fix

Always read at least 2 rollout transcripts (1 pass, 1 fail) before tuning.

Specialized diagnostic roles

Role 1: Rollout Analyst

Q: What did agents actually do, and why pass/fail? Run the analysis script (from maddy-iteration-loop Phase 3), read ≥2 transcripts (≥1 pass, ≥1 fail), classify the distribution (deadweight / healthy / correlated). Output one paragraph; no fixes yet.

Role 2: Setup Drift Specialist

Q: Of the three drift sources, which are actually biting? For each (kube-system reconciler, bleater-env reconciler, canary config-pinner) check whether failing transcripts mention discovering/stopping it. The bleater-env one is usually the most-missed (innocuous name + namespace). Output: ranked variance impact. Prefer setup.sh edits.

Role 3: Consistency Specialist

Q: What controls delivery_consistency variance independent of governance? Inspect: do failing agents stop all drift but forget to scale maddy to 2? Is the legacy replica re-spawning inside the window? The decorrelating lever is the convergence requirement (initial replicas=1). Output: a setup.sh/grader edit that moves consistency without touching governance discoverability.

Role 4: Governance Specialist

Q: What controls observability_governance variance independent of consistency? The three artifacts: POD_NAME env, maddy-gitops-source edit, per-pod PromQL rule. Read failing transcripts: did agents write a ConfigMap with "pod" in a comment but no by (pod) expr (Layer-3 miss)? Did they skip the downward API? First rule out bleater-env reachability as a structural deadweight. Output: a task.yaml hint tweak or a Layer-3 strictness change (within reason).

Role 5: Coupling Reviewer (the maddy-critical role)

Q: Does the proposed edit move ONLY the shared drift surface? Walk the inventory:

• Drift cadence/discoverability → moves BOTH subscores (correlation risk). Only acceptable when the explicit goal is shifting avg.
• Initial maddy replicas → consistency-only (safe to decorrelate)
• The three observability artifacts → governance-only (safe to decorrelate)
• wait_consistent on one subscore → affects only that subscore's window (drift restart churn can flake both if both windows shrink) Output: "no correlation risk — moves only X" OR "REJECT — this is a shared-surface lever; it will move both subscores; propose a decorrelating alternative."

Role 6: QC Reviewer

Q: Would this edit break local quality or add a permanent fail? Expected: 18 substantive checks pass; the v1_feedback section FAILs as accepted noise (heredoc-invisible baseline config + annotation-based discovery). Reject: naming a drift controller / bleater.io/purpose / bleater-env / config-pinner in task.yaml or visible setup; requiring an exact ConfigMap name in the per-pod-rule check. Output: "QC delta 0/+1" or "REJECT — reviewer will flag Y."

Role 7: Final Recommender

Produce exactly this 8-point block:

1. Root cause (one sentence)
2. Why avg shifted (the score math)
3. Smallest safe fix (one edit, one file)
4. Expected rollout impact (projected avg + per-subscore distribution + correlation)
5. Expected QC impact (delta on the 18-check baseline)
6. Risk assessment (ranked)
7. Exact files to modify
8. Exact lines/components affected

Only after the Recommender's block does the orchestrator (maddy-iteration-loop) hand off to maddy-task-guardian for the edit + push.

Operating procedure — diagnose, propose, approve, apply

NEVER edit immediately on a fresh diagnosis. Walk in order:

1. Confirm trigger — Oracle failure → hand to maddy-validation-debugger. This skill is rollout regressions only.
2. Read — grader.py, setup.sh, task.yaml, latest rollout JSONs, ≥2 transcripts (1 pass, 1 fail).
3. Diagnose — Role 1 for the distribution; look up the symptom; decide if it's deadweight, correlation collapse, grader strictness, or drift intensity.
4. Pick the lever — Level 1 (setup.sh) unless exhausted; Level 2 (grader) only if necessary; Level 3 (task.yaml) last. Prefer a decorrelating lever unless intentionally shifting avg.
5. Refine — adopt the matching specialist role (Consistency / Governance / Setup Drift).
6. Review — Role 5 (Coupling) and Role 6 (QC). If either rejects, return to step 4.
7. Recommend — Role 7's 8-point block.
8. NEVER apply from this skill. Hand to maddy-iteration-loop → maddy-task-guardian for syntax check, push, validation.

Hard constraints (same as task-guardian)

What is NOT graded (do not add these to grader)

• Exact ConfigMap names (per-pod rule is name-agnostic by design)
• Replica count on the drift Deployments beyond ==0
• The egress NetworkPolicy contents (scenario realism, not graded)
• maddy.conf fields beyond the relay_mode = smarthost / not-direct markers
• Annotation/label values that the baseline documents (copy-from-baseline dead-weight)

Preferred outcomes

Healthy: 0.0 / 0.5 / 1.0 all represented across a 5-rollout batch; avg –0.55; both subscores vary AND are not perfectly correlated; failures are legitimate (a discoverable drift source missed, or a vague per-pod rule, or forgot to scale to 2).

Unhealthy: any subscore deadweight; avg > ; the two subscores correlate perfectly every batch (correlation collapse); a structural impossibility (bleater-env unreachable); static-only grading creeping in.

If you observe an Unhealthy pattern, halt and write a summary rather than pushing another fix. Two cycles chasing the same Unhealthy pattern means the lever choice was wrong.