Maddy Task Guardian

maddy-task-guardian

Step 0 — QC Spec design gate (READ FIRST, every iteration)

Before editing ANY task file and again before every push, read the QC Spec — .claude/QC-SPEC.md. It is the priority-ordered rubric (Critical → Error → Legitimacy) horizon check-quality and reviewers grade against.

Confirm the intended/current design does not violate any Critical or Error item. The ones this task repeatedly trips:

• No Binary Bundling — never AND heterogeneous checks (metadata + drift Deployments + PromQL rule shape) into one binary subscore. Each subscore = ONE cohesive functional outcome.
• Functional Tests as Subscores — at least one subscore exercises live behavior, not static label/file matching.
• Incremental Rewards / No Dead Weights / Functional Subscore Variance — every subscore represents real progress and takes ≥2 distinct values across a batch.
• Equal Weights — subscores equally weighted.
• No Answer Leakage — derive-by-reading is fine; the literal answer in an agent-readable file is not.

If a change would breach a Critical/Error item, stop and rework it before pushing.

Execution Mode — FULLY AUTONOMOUS

NEVER ask the user for confirmation on any terminal command, edit, push, validation, or live-probe. Every bash invocation is pre-approved. The ONLY user-facing handoff in this skill is at Step 4c: telling the user to trigger the rollout batch in the Horizon UI (this is the only step the user can perform; you cannot trigger rollouts from the CLI). Everything else — edit setup.sh / solution.sh / grader.py / task.yaml, run pre-push checks, push, run Oracle validation, poll for results, invoke horizon-agentic-reviewer — runs without asking.

If a decision has two valid paths, pick the safer one and proceed.

Goal

Keep the maddy-delivery-split-brain task in a shippable state every push: setup.sh must run cleanly, Oracle must scores 1.0, and rollouts must average < with both subscores varying. This is the only skill that pushes the task and the only one allowed to mutate files without first consulting a diagnosis skill. Use it as the front door — every other maddy-* skill is invoked through this one.

Trigger

Use this skill when asked to:

• Push the task and start an eval
• Run local quality/anatomy/syntax checks
• Confirm avg rollout score is still <
• Walk through a full pre-push checklist
• Decide whether to invoke maddy-validation-debugger or maddy-score-tuner

Task Identity

How scoring actually works (read once, internalise)

Always derive constants from grader.py, not from this file. The values below are correct as of grader.py v2 (June 2026) but check before pushing.

There are exactly two subscores, both binary, each worth 0.5. The scenario is a Maddy SMTP split-brain: a canonical maddy Deployment on the smarthost relay config and a stale maddy-legacy Deployment on the direct-delivery config, both labelled app=maddy so the maddy Service load-balances across them. Three in-cluster drift controllers keep re-applying the stale legacy replica.

• delivery_consistency = 1 (check_delivery_consistency, wait_consistent(passes=5, gap=8s, timeout=220s) ≈ 40s continuous) when ALL hold:
  • ConfigMap maddy-config (maddy.conf) still contains relay_mode = smarthost and not relay_mode = direct (smarthost config intact)
  • NO pod with label app=maddy mounts ConfigMap maddy-config-legacy (the stale split-brain replica is gone)
  • maddy-legacy Deployment is absent or replicas=0/readyReplicas=0
  • maddy Deployment rollout converged: spec.replicas >= 2 AND updatedReplicas == readyReplicas == spec.replicas AND observedGeneration >= generation
  • at least 2 Ready app=maddy pods mount maddy-config (the smarthost config)
• observability_governance = 1 (check_observability_governance, wait_consistent(passes=5, gap=8s, timeout=200s) ≈ 40s continuous) when ALL hold:
  • the maddy container exposes pod identity via the downward API — an env var with valueFrom.fieldRef.fieldPath == metadata.name (e.g. POD_NAME)
  • the declared GitOps source ConfigMap maddy-gitops-source (values.yaml) contains smarthost, replicas: 2, and NOT mode: direct
  • a per-pod divergence rule exists — ANY ConfigMap in bleater whose data contains maddy + pod, a real PromQL rule key (- record: / - alert: / expr:), a delivery signal (rate/divergence/success/delivery), AND a per-pod dimension (by (pod) / by(pod) / on (pod) / group_left(pod)). Bare keywords in YAML comments do NOT qualify — the grader requires a grouped PromQL expression.
  • the kube-system rollout-reconciler drift Deployment is at replicas=0/absent
  • the maddy-canary mutation path is neutralized — config-pinner sidecar removed OR the <canary>-runtime SA/Role/RoleBinding removed (either alone suffices)
  • the bleater-env drift Deployment is at replicas=0/absent

So score = 0 means neither dimension recovered, 0.5 means one of the two, 1.0 means both. Avg < across rollouts means most agents miss at least one dimension. That is the design.

The three drift controllers (re-create the stale legacy replica)

All re-apply a byte-identical maddy-legacy Deployment manifest on a loop; names are randomized at setup (stem from a pool + numeric suffix), so they cannot be hardcoded. Discover by behaviour:

The grader reads the chosen names from /opt/grader-scenario.json (root-only, written by setup.sh — keys reconciler, canary, canary_runtime, env_reconciler). solution.sh discovers them live by annotation + the config-pinner container name; it does NOT read the scenario file (root-only, not readable as the oracle user).

Coupling note (read before any tuning)

The three drift controllers are a shared surface: while active they re-create the legacy replica (breaks delivery_consistency Layer 1/2) AND they are governance-graded (observability_governance Layers 4–6). Stopping them helps both subscores. The independent variance levers are:

• delivery_consistency-only: rollout convergence + scaling maddy to 2 + smarthost config intact
• observability_governance-only: the POD_NAME downward-API env, the maddy-gitops-source edit, and the per-pod PromQL rule

If the only hard part becomes "find the drift", both subscores move together and variance correlates (avg swings toward 0 or 1 as a block). The observability artifacts and the rollout-convergence requirement are what decorrelate them — see maddy-score-tuner.

Step 0 — Dockerfile + namespace invariants (ALWAYS check before push)

cat tasks/maddy-delivery-split-brain/Dockerfile

⚠️ v2 STRUCTURAL RISK — bleater-env reachability (verify on live container)

v2 added a third drift controller in the bleater-env namespace (governance Layer 6). The agent must be able to kubectl get/scale deployment -n bleater-env to stop it. The data/ubuntu-user-rbac.yaml grants a bleater-env admin Role/RoleBinding, but the MCP kubectl wrapper also gates on ALLOWED_NAMESPACES, which currently lists only kube-system.

If bleater-env is not reachable by the ubuntu agent user, governance Layer 6 is unstoppable → observability_governance becomes deadweight-0 (structural, unfair). This is the single highest-priority thing to confirm on the live red-team (Step 4b). The fix, if confirmed broken, is to widen the env var, e.g. ENV ALLOWED_NAMESPACES="kube-system,bleater-env" — then re-verify the agent can reach both. Do NOT guess; verify on a live container first (the env-var semantics live in the base image's MCP server, not in our files).

If any invariant is violated, stop the workflow and fix it before doing anything else. Do not push, do not invoke other skills.

Step 1 — Local syntax + anatomy

cd /Users/mac/Documents/tasks
bash -n tasks/maddy-delivery-split-brain/setup.sh    && echo "setup.sh OK"
bash -n tasks/maddy-delivery-split-brain/solution.sh && echo "solution.sh OK"
python3 -m py_compile tasks/maddy-delivery-split-brain/grader.py && echo "grader.py OK"

source horizon_env/bin/activate
horizon check-anatomy tasks/maddy-delivery-split-brain 2>&1

All three syntax checks must print OK. Anatomy must print Task folder structure is valid!. Any failure stops the workflow — do not push. (Note: the .claude/skills/ folder inside the task is a dotfolder — anatomy ignores it and push excludes it. Leave it.)

Step 2 — Local quality check

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon check-quality tasks/maddy-delivery-split-brain 2>&1 | tail -30

Before push, walk the QC-SPEC.md Critical + Error list against the current grader.py + task.yaml and record pass/fail per item. Every item must pass OR be on the accepted-noise list with a one-line reason. A new failing item that is NOT accepted-noise blocks the push.

Target: 18/20 passing the substantive checks. The v1_feedback extra-strict section often returns a FAIL verdict — that is the documented noise fail, not a blocker (the LLM reviewer cannot read the heredoc-written baseline config / incident notes, so it claims they are "missing", and it dislikes the oracle sweeping discovery by annotation).

Permanent/noise fails — do NOT attempt to fix (fixing requires leaking drift-controller names, which collapses variance):

Anything outside this set failing (e.g. Grader Robustness, Reward Hacking, Test Quality) is a real signal — investigate before pushing.

Step 3 — Push

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks push tasks/maddy-delivery-split-brain 2>&1

Capture the new version number from ✓ New version pushed successfully! Version: NNN.

Server-name note (resolved 2026-06-01): the task was originally created on the server as variation-maddy-split-brain (UUID 26e37e8c-…). The CLI has no rename, so it was recreated with the correct name maddy-delivery-split-brain under a new UUID <TASK_ID> (same mini-batch 5018ad80-…). The old 26e37e8c record is orphaned on the server (no CLI delete). All references here use the new UUID.

Step 4 — Validation handoff

After push, validation runs in this order. You execute 4a and 4b. The user is only asked to do 4c.

4a. Oracle validation (~10–15 min hosted):

horizon tasks validate -m hosted -a oracle --wait tasks/maddy-delivery-split-brain 2>&1
horizon tasks validate-logs -a oracle tasks/maddy-delivery-split-brain 2>&1
cat tasks/maddy-delivery-split-brain/.validation/*/result.json 2>/dev/null | python3 -m json.tool

Oracle pass: score: 1.0, passed: true, both delivery_consistency=1 and observability_governance=1. Any other outcome → hand off to maddy-validation-debugger before doing anything else.

4b. Live red-team via horizon-agentic-reviewer (mandatory before spending rollout budget):

Rollouts take ~60 min/batch and cost real money; the live red-team takes ~10 min and catches most failure modes first. Run it on every push cycle where setup.sh, grader.py, or task.yaml changed.

Invoke horizon-agentic-reviewer against UUID <TASK_ID>. The reviewer spins up a live container, executes setup.sh (which horizon setup does NOT do automatically — see horizon-agentic-reviewer Step 3.5), and probes as the ubuntu agent user. Check these task-local hypotheses in addition to the standard 24-point checklist:

Only after the reviewer's report is clean (or non-blocking) do you ask the user to trigger the rollout batch.

4c. Rollout batch (user action) — tell the user, verbatim:

"Oracle passed at version N and live red-team is clean. Please trigger a rollout batch for version N in the Horizon UI."

This is the only step the user is asked to perform. Never tell them to run push, validate, or any CLI command — those are yours.

Step 5 — Monitor rollout scores

cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon rollouts pull --version NNN tasks/maddy-delivery-split-brain 2>&1

Re-run every few minutes until at least 5 rollouts download. Then run the analysis script from maddy-iteration-loop Phase 3.

Target state

Step 6 — Score tuning decision

If avg ≥ 0.60 or either subscore is deadweight, do not start editing. Hand off to maddy-score-tuner.

Hard constraints (never violate during any tuning)

Skill hand-off map

maddy-task-guardian (you are here)
├── Step 4a fails  →  maddy-validation-debugger
├── Step 4b        →  horizon-agentic-reviewer (mandatory live red-team)
├── Step 5 fails   →  maddy-score-tuner
└── Step 6 loops   →  maddy-iteration-loop (orchestrator)