# Maddy Task Guardian

> End-to-end maintenance skill for the maddy-delivery-split-brain task (<TASK_ID>). Owns the pre-push safety checks, local quality/anatomy gates, push, Oracle validation handoff, and rollout monitoring. Primary goal is to keep avg rollout score < <TARGET_MEAN> with healthy variance across both subscores. Calls maddy-validation-debugger when Oracle fails and maddy-score-tuner when rollouts regress.

- Skill: `omonuj/maddy-task-guardian` (Agent Skill)
- Install (CLI): `npx skillmds@latest add omonuj/maddy-task-guardian`
- Raw SKILL.md: https://api.skillmd.com/api/skills/omonuj/maddy-task-guardian/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: omonuj (https://skillmd.com/u/omonuj)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/omonuj/maddy-task-guardian

---


# maddy-task-guardian

## Step 0 — QC Spec design gate (READ FIRST, every iteration)

Before editing ANY task file and again before every push, read the QC Spec — `.claude/QC-SPEC.md`. It is the priority-ordered rubric (Critical → Error → Legitimacy) `horizon check-quality` and reviewers grade against.

Confirm the intended/current design does not violate any Critical or Error item. The ones this task repeatedly trips:
- **No Binary Bundling** — never AND heterogeneous checks (metadata + drift Deployments + PromQL rule shape) into one binary subscore. Each subscore = ONE cohesive functional outcome.
- **Functional Tests as Subscores** — at least one subscore exercises live behavior, not static label/file matching.
- **Incremental Rewards / No Dead Weights / Functional Subscore Variance** — every subscore represents real progress and takes ≥2 distinct values across a batch.
- **Equal Weights** — subscores equally weighted.
- **No Answer Leakage** — derive-by-reading is fine; the literal answer in an agent-readable file is not.

If a change would breach a Critical/Error item, stop and rework it before pushing.

## Execution Mode — FULLY AUTONOMOUS

**NEVER ask the user for confirmation on any terminal command, edit, push, validation, or live-probe.** Every bash invocation is pre-approved. The ONLY user-facing handoff in this skill is at Step 4c: telling the user to trigger the rollout batch in the Horizon UI (this is the only step the user can perform; you cannot trigger rollouts from the CLI). Everything else — edit setup.sh / solution.sh / grader.py / task.yaml, run pre-push checks, push, run Oracle validation, poll for results, invoke horizon-agentic-reviewer — runs without asking.

If a decision has two valid paths, pick the safer one and proceed.

## Goal

Keep the maddy-delivery-split-brain task in a shippable state every push: setup.sh must run cleanly, Oracle must scores 1.0, and rollouts must average < <TARGET_MEAN> with both subscores varying. This is the only skill that pushes the task and the only one allowed to mutate files without first consulting a diagnosis skill. Use it as the front door — every other maddy-* skill is invoked through this one.

## Trigger

Use this skill when asked to:
- Push the task and start an eval
- Run local quality/anatomy/syntax checks
- Confirm avg rollout score is still < <TARGET_MEAN>
- Walk through a full pre-push checklist
- Decide whether to invoke maddy-validation-debugger or maddy-score-tuner

## Task Identity

| Field | Value |
|---|---|
| **Task UUID** | `<TASK_ID>` |
| **Task slug** | `maddy-delivery-split-brain` |
| **Local task path** | `tasks/maddy-delivery-split-brain/` |
| **Horizon root** | `/Users/mac/Documents/tasks` |
| **Venv activate** | `source /Users/mac/Documents/tasks/horizon_env/bin/activate` |
| **Scoring** | `delivery_consistency × 0.5 + observability_governance × 0.5` — both binary |
| **Target avg** | rollout `avg < <TARGET_MEAN> with both subscores varying (neither always 0 nor always 1) |
| **Mini-batch** | `<TASK_ID>` |

## How scoring actually works (read once, internalise)

**Always derive constants from `grader.py`, not from this file.** The values below are correct as of grader.py v2 (June 2026) but check before pushing.

There are exactly two subscores, both binary, each worth 0.5. The scenario is a Maddy SMTP split-brain: a canonical `maddy` Deployment on the **smarthost** relay config and a stale `maddy-legacy` Deployment on the **direct-delivery** config, both labelled `app=maddy` so the `maddy` Service load-balances across them. Three in-cluster drift controllers keep re-applying the stale legacy replica.

- **`delivery_consistency = 1`** (`check_delivery_consistency`, `wait_consistent(passes=5, gap=8s, timeout=220s)` ≈ 40s continuous) when ALL hold:
  - ConfigMap `maddy-config` (`maddy.conf`) still contains `relay_mode = smarthost` and not `relay_mode = direct` (smarthost config intact)
  - NO pod with label `app=maddy` mounts ConfigMap `maddy-config-legacy` (the stale split-brain replica is gone)
  - `maddy-legacy` Deployment is absent or `replicas=0`/`readyReplicas=0`
  - `maddy` Deployment rollout converged: `spec.replicas >= 2` AND `updatedReplicas == readyReplicas == spec.replicas` AND `observedGeneration >= generation`
  - at least `2` Ready `app=maddy` pods mount `maddy-config` (the smarthost config)
- **`observability_governance = 1`** (`check_observability_governance`, `wait_consistent(passes=5, gap=8s, timeout=200s)` ≈ 40s continuous) when ALL hold:
  - the `maddy` container exposes pod identity via the downward API — an env var with `valueFrom.fieldRef.fieldPath == metadata.name` (e.g. `POD_NAME`)
  - the declared GitOps source ConfigMap `maddy-gitops-source` (`values.yaml`) contains `smarthost`, `replicas: 2`, and NOT `mode: direct`
  - a per-pod divergence rule exists — ANY ConfigMap in `bleater` whose data contains `maddy` + `pod`, a real PromQL rule key (`- record:` / `- alert:` / `expr:`), a delivery signal (`rate`/`divergence`/`success`/`delivery`), AND a per-pod dimension (`by (pod)` / `by(pod)` / `on (pod)` / `group_left(pod)`). **Bare keywords in YAML comments do NOT qualify — the grader requires a grouped PromQL expression.**
  - the kube-system rollout-reconciler drift Deployment is at `replicas=0`/absent
  - the `maddy-canary` mutation path is neutralized — `config-pinner` sidecar removed OR the `<canary>-runtime` SA/Role/RoleBinding removed (either alone suffices)
  - the bleater-env drift Deployment is at `replicas=0`/absent

So `score = 0` means neither dimension recovered, `0.5` means one of the two, `1.0` means both. Avg < <TARGET_MEAN> across rollouts means most agents miss at least one dimension. That is the design.

### The three drift controllers (re-create the stale legacy replica)

All re-apply a byte-identical `maddy-legacy` Deployment manifest on a loop; **names are randomized** at setup (stem from a pool + numeric suffix), so they cannot be hardcoded. Discover by behaviour:

| Drift source | Namespace | Cadence | How to discover | Scenario-file key |
|---|---|---|---|---|
| kube-system reconciler | `kube-system` | 5s | annotation `bleater.io/purpose: "maddy rollout convergence"` | `reconciler` |
| bleater-env reconciler | `bleater-env` | 20s | annotation `bleater.io/purpose: "maddy rollout convergence"` (innocuous non-maddy name) | `env_reconciler` |
| canary `config-pinner` | `bleater` | 11s | a container named `config-pinner` in a Deployment; runtime RBAC at `<canary>-runtime` | `canary` / `canary_runtime` |

The grader reads the chosen names from `/opt/grader-scenario.json` (root-only, written by setup.sh — keys `reconciler`, `canary`, `canary_runtime`, `env_reconciler`). solution.sh discovers them live by annotation + the `config-pinner` container name; it does NOT read the scenario file (root-only, not readable as the oracle user).

### Coupling note (read before any tuning)

The three drift controllers are a **shared surface**: while active they re-create the legacy replica (breaks `delivery_consistency` Layer 1/2) AND they are governance-graded (`observability_governance` Layers 4–6). Stopping them helps **both** subscores. The *independent* variance levers are:
- `delivery_consistency`-only: rollout convergence + scaling `maddy` to 2 + smarthost config intact
- `observability_governance`-only: the `POD_NAME` downward-API env, the `maddy-gitops-source` edit, and the per-pod PromQL rule

If the only hard part becomes "find the drift", both subscores move together and variance correlates (avg swings toward 0 or 1 as a block). The observability artifacts and the rollout-convergence requirement are what decorrelate them — see `maddy-score-tuner`.

---

## Step 0 — Dockerfile + namespace invariants (ALWAYS check before push)

```bash
cat tasks/maddy-delivery-split-brain/Dockerfile
```

| Invariant | Required state | Why — what breaks if violated |
|---|---|---|
| `ENABLE_ISTIO_BLEATER` | **must NOT appear** | Triggers k3s 17+ pod force-deletes on startup. k3s fails `not ready after 180s`, grader returns `score: 0, feedback: null`. Setup crash, not gradeable. |
| `ENV ALLOWED_NAMESPACES` | **must be present** and **must include `kube-system`** | Without it, the kube-system reconciler can't be stopped → `observability_governance` Layer 4 structurally always-0. |
| `COPY data/ubuntu-user-rbac.yaml /mcp_server/Nebula/infra/k8s/rbac/ubuntu-user-rbac.yaml` | **must be present**, path under `data/` | `kubectl get nodes` in the `DO NOT CHANGE` block of setup.sh requires the node-access ClusterRole. Without the COPY, setup aborts before fault injection. |

### ⚠️ v2 STRUCTURAL RISK — bleater-env reachability (verify on live container)

v2 added a **third** drift controller in the `bleater-env` namespace (governance Layer 6). The agent must be able to `kubectl get/scale deployment -n bleater-env` to stop it. The `data/ubuntu-user-rbac.yaml` grants a `bleater-env` admin Role/RoleBinding, **but the MCP `kubectl` wrapper also gates on `ALLOWED_NAMESPACES`**, which currently lists only `kube-system`.

**If `bleater-env` is not reachable by the `ubuntu` agent user, governance Layer 6 is unstoppable → `observability_governance` becomes deadweight-0 (structural, unfair).** This is the single highest-priority thing to confirm on the live red-team (Step 4b). The fix, if confirmed broken, is to widen the env var, e.g. `ENV ALLOWED_NAMESPACES="kube-system,bleater-env"` — then re-verify the agent can reach both. Do NOT guess; verify on a live container first (the env-var semantics live in the base image's MCP server, not in our files).

If any invariant is violated, **stop the workflow and fix it before doing anything else.** Do not push, do not invoke other skills.

---

## Step 1 — Local syntax + anatomy

```bash
cd /Users/mac/Documents/tasks
bash -n tasks/maddy-delivery-split-brain/setup.sh    && echo "setup.sh OK"
bash -n tasks/maddy-delivery-split-brain/solution.sh && echo "solution.sh OK"
python3 -m py_compile tasks/maddy-delivery-split-brain/grader.py && echo "grader.py OK"

source horizon_env/bin/activate
horizon check-anatomy tasks/maddy-delivery-split-brain 2>&1
```

All three syntax checks must print `OK`. Anatomy must print `Task folder structure is valid!`. Any failure stops the workflow — do not push. (Note: the `.claude/skills/` folder inside the task is a dotfolder — anatomy ignores it and push excludes it. Leave it.)

## Step 2 — Local quality check

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon check-quality tasks/maddy-delivery-split-brain 2>&1 | tail -30
```

**Before push, walk the QC-SPEC.md Critical + Error list against the current `grader.py` + `task.yaml`** and record pass/fail per item. Every item must pass OR be on the accepted-noise list with a one-line reason. A new failing item that is NOT accepted-noise blocks the push.

**Target: 18/20 passing the substantive checks. The `v1_feedback` extra-strict section often returns a `FAIL` verdict — that is the documented noise fail, not a blocker** (the LLM reviewer cannot read the heredoc-written baseline config / incident notes, so it claims they are "missing", and it dislikes the oracle sweeping discovery by annotation).

Permanent/noise fails — do NOT attempt to fix (fixing requires leaking drift-controller names, which collapses variance):

| Check / section | Why it permanently fails — and why we accept it |
|---|---|
| `v1_feedback` | Reviewer hallucinates "missing" baseline config & incident notes (they are written via heredoc inside setup.sh, invisible to static review) and objects that solution.sh discovers drift by annotation. Both are by design. |
| `behavior_in_task_documentation` / `task_clarity` (intermittent) | Same root cause — reviewer can't read the heredoc baseline config. Listing the drift controller names in `task.yaml` would collapse `observability_governance` to always-1. |

Anything **outside** this set failing (e.g. Grader Robustness, Reward Hacking, Test Quality) is a real signal — investigate before pushing.

## Step 3 — Push

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks push tasks/maddy-delivery-split-brain 2>&1
```

Capture the new version number from `✓ New version pushed successfully! Version: NNN`.

> Server-name note (resolved 2026-06-01): the task was originally created on the server as `variation-maddy-split-brain` (UUID `26e37e8c-…`). The CLI has no rename, so it was **recreated** with the correct name `maddy-delivery-split-brain` under a new UUID `<TASK_ID>` (same mini-batch `5018ad80-…`). The old `26e37e8c` record is orphaned on the server (no CLI delete). All references here use the new UUID.

## Step 4 — Validation handoff

After push, validation runs in this order. **You execute 4a and 4b. The user is only asked to do 4c.**

**4a. Oracle validation** (~10–15 min hosted):
```bash
horizon tasks validate -m hosted -a oracle --wait tasks/maddy-delivery-split-brain 2>&1
horizon tasks validate-logs -a oracle tasks/maddy-delivery-split-brain 2>&1
cat tasks/maddy-delivery-split-brain/.validation/*/result.json 2>/dev/null | python3 -m json.tool
```

Oracle pass: `score: 1.0`, `passed: true`, both `delivery_consistency=1` and `observability_governance=1`. **Any other outcome → hand off to `maddy-validation-debugger` before doing anything else.**

**4b. Live red-team via `horizon-agentic-reviewer`** (mandatory before spending rollout budget):

Rollouts take ~60 min/batch and cost real money; the live red-team takes ~10 min and catches most failure modes first. Run it on every push cycle where setup.sh, grader.py, or task.yaml changed.

Invoke `horizon-agentic-reviewer` against UUID `<TASK_ID>`. The reviewer spins up a live container, executes setup.sh (which `horizon setup` does NOT do automatically — see horizon-agentic-reviewer Step 3.5), and probes as the `ubuntu` agent user. Check these task-local hypotheses in addition to the standard 24-point checklist:

| Hypothesis | Live probe (as `ubuntu`, the agent role) |
|---|---|
| **bleater-env is reachable** (the #1 risk) | `docker exec -u ubuntu apex-arena-<uuid> bash -c 'kubectl get deploy -n bleater-env'` — must NOT be `Forbidden`. If forbidden, governance Layer 6 is structurally unsolvable → fix `ALLOWED_NAMESPACES`. |
| Split-brain is live | `kubectl get pods -n bleater -l app=maddy -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.volumes[*].configMap.name}{"\n"}{end}'` — at least one pod must mount `maddy-config` and at least one `maddy-config-legacy` |
| kube-system reconciler exists + annotated | `kubectl get deploy -n kube-system -o json \| python3 -c "import json,sys; [print(d['metadata']['name']) for d in json.load(sys.stdin)['items'] if (d['metadata'].get('annotations') or {}).get('bleater.io/purpose')=='maddy rollout convergence']"` — must print one name |
| bleater-env reconciler exists + annotated | same one-liner against `-n bleater-env` — must print one name |
| canary config-pinner + runtime RBAC exist | `kubectl get deploy -n bleater -o json \| grep config-pinner` AND `kubectl get sa,role,rolebinding -n bleater \| grep -- -runtime` |
| Drift controller names are NOT leaked in baseline | `grep -iE "reconciler\|config-pinner\|canary\|convergence\|bleater-env" /home/ubuntu/maddy-delivery-baseline-config.txt` — should NOT reveal the live randomized names |
| Stale GitOps source is in place | `kubectl get configmap maddy-gitops-source -n bleater -o jsonpath='{.data.values\.yaml}'` — must show `replicas: 1` and `mode: direct` |
| A no-op agent cannot pass either subscore | Reset, leave alone, run the grader from the reviewer's bucket-C probe — both subscores must be 0 |
| Setup hygiene: events cleared | `kubectl get events -A --no-headers \| wc -l` — should be near 0 right after setup (so agents can't reverse-engineer faults) |

**Only after the reviewer's report is clean (or non-blocking) do you ask the user to trigger the rollout batch.**

**4c. Rollout batch** (user action) — tell the user, verbatim:
> "Oracle passed at version N and live red-team is clean. Please trigger a rollout batch for version N in the Horizon UI."

This is the **only** step the user is asked to perform. Never tell them to run push, validate, or any CLI command — those are yours.

## Step 5 — Monitor rollout scores

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon rollouts pull --version NNN tasks/maddy-delivery-split-brain 2>&1
```

Re-run every few minutes until at least 5 rollouts download. Then run the analysis script from `maddy-iteration-loop` Phase 3.

### Target state

| Metric | Target | Reason |
|---|---|---|
| avg rollout score | **< 0.60** | Nebula creator workflow requirement |
| `delivery_consistency` outcomes | both `0` and `1` appear | Subscore must vary — never deadweight |
| `observability_governance` outcomes | both `0` and `1` appear | Subscore must vary — never deadweight |
| `delivery_consistency` pass rate | roughly 20–60% | Above 60% → drift too easy to find OR rollout-convergence too forgiving; below 20% → a drift source is unstoppable (re-check bleater-env reachability) |
| `observability_governance` pass rate | roughly 20–50% | Above 50% → per-pod rule / discovery too easy; below 20% → structural (bleater-env reachability, or the per-pod-rule gate too strict) |

## Step 6 — Score tuning decision

If avg ≥ 0.60 or either subscore is deadweight, **do not start editing.** Hand off to `maddy-score-tuner`.

| Observation | First hand-off |
|---|---|
| Oracle fails (any score < 1.0) | `maddy-validation-debugger` |
| Rollout avg ≥ 0.60 OR any subscore deadweight | `maddy-score-tuner` |
| `observability_governance` stuck at 0 across all rollouts, Oracle passes | `maddy-score-tuner` (suspect bleater-env reachability first) |
| Either subscore stuck at 1 across all rollouts | `maddy-score-tuner` (discoverability too high) |
| Agent-facing env doesn't match what setup.sh claims | `horizon-agentic-reviewer` |

---

## Hard constraints (never violate during any tuning)

| Constraint | Why — incident or design intent |
|---|---|
| Do NOT add `ENABLE_ISTIO_BLEATER=true` | k3s startup failure → `score: 0, feedback: null`. Unfixable from the task side. |
| Do NOT remove `kube-system` from `ALLOWED_NAMESPACES` | Kills the kube-system reconciler-stop path → governance Layer 4 always-0. |
| Do NOT remove `bleater-env` reachability once added | Governance Layer 6 needs it; without it that layer is unsolvable. |
| Do NOT change the `COPY data/ubuntu-user-rbac.yaml` path | The `DO NOT CHANGE` block in setup.sh depends on this exact path; setup aborts. |
| Do NOT list drift-controller names (or the `bleater.io/purpose` value) in `task.yaml` or the baseline-config heredoc | Collapses `observability_governance` to always-1 — discovery is the variance axis. |
| Do NOT use external registry images for any drift source | Cluster is air-gapped. Use `${SIDECAR_IMAGE}` (an in-cluster image). |
| Do NOT weaken the per-pod-rule check to accept bare keywords | The `by (pod)` + PromQL-key requirement is what stops agents passing with a vague ConfigMap; it is signal, not pedantry. But do NOT require an exact ConfigMap name either (name-agnostic by design). |
| Do NOT tighten `wait_consistent` past `passes=5, gap=8s` (consistency `timeout=220s`, governance `timeout=200s`) | Above ceiling, legitimate solutions flake into dead-zero (drift restart churn). |
| Do NOT simplify solution.sh to make Oracle easier | solution.sh is the oracle standard, not a tuning lever. If Oracle fails, the bug is in setup.sh/grader.py. |

---

## Skill hand-off map

```
maddy-task-guardian (you are here)
├── Step 4a fails  →  maddy-validation-debugger
├── Step 4b        →  horizon-agentic-reviewer (mandatory live red-team)
├── Step 5 fails   →  maddy-score-tuner
└── Step 6 loops   →  maddy-iteration-loop (orchestrator)
```