# Mtls Task Guardian

> End-to-end maintenance skill for the variation-fanout-pipeline-break task (<TASK_ID>). Owns the pre-push safety checks, local quality/anatomy gates, push, Oracle validation handoff, and rollout monitoring. Primary goal is to keep avg rollout score < <TARGET_MEAN> with healthy variance across both subscores. Calls mtls-validation-debugger when Oracle fails and mtls-score-tuner when rollouts regress.

- Skill: `omonuj/mtls-task-guardian` (Agent Skill)
- Install (CLI): `npx skillmds@latest add omonuj/mtls-task-guardian`
- Raw SKILL.md: https://api.skillmd.com/api/skills/omonuj/mtls-task-guardian/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: omonuj (https://skillmd.com/u/omonuj)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/omonuj/mtls-task-guardian

---


# mtls-task-guardian

## Step 0 — QC Spec design gate (READ FIRST, every iteration)

Before editing ANY task file and again before every push, read the QC Spec — `.claude/QC-SPEC.md` (kept inside the `.claude/` dotfolder so it is excluded from `check-anatomy` and never bundled into the eval). It is the priority-ordered rubric (Critical → Error → Legitimacy) `horizon check-quality` and reviewers grade against.

Confirm the intended/current design does not violate any Critical or Error item. The ones this task repeatedly trips:
- **No Binary Bundling** — `trust_governance` must NOT AND together heterogeneous checks (metadata labels + drift Deployments + RBAC). Each subscore = ONE cohesive functional outcome.
- **Functional Tests as Subscores** — at least one subscore exercises live behavior (the handshake), not static label/file matching.
- **Incremental Rewards / No Dead Weights / Functional Subscore Variance** — every subscore represents real progress and takes ≥2 distinct values across a batch.
- **Equal Weights** — both subscores at 0.5.
- **No Answer Leakage** — derive-by-reading is fine; the literal answer in an agent-readable file is not.

If a change would breach a Critical/Error item, stop and rework it — do not push with a recorded justification unless it is a documented accepted-noise exception (see Step 2).

## Execution Mode — FULLY AUTONOMOUS

**NEVER ask the user for confirmation on any terminal command, edit, push, validation, or live-probe.** Every bash invocation is pre-approved. The ONLY user-facing handoff in this skill is at Step 4c: telling the user to trigger the rollout batch in the Horizon UI (this is the only step the user can perform; you cannot trigger rollouts from the CLI). Everything else — edit setup.sh / solution.sh / grader.py / task.yaml, run pre-push checks, push, run Oracle validation, poll for results, invoke horizon-agentic-reviewer — runs without asking.

If a decision has two valid paths, pick the safer one and proceed.

## Goal

Keep the variation-fanout-pipeline-break task in a shippable state every push: setup.sh must run cleanly, Oracle must scores 1.0, and rollouts must average < <TARGET_MEAN> with both subscores varying. This is the only skill that pushes the task and the only one allowed to mutate files without first consulting a diagnosis skill. Use it as the front door — every other mtls-* skill is invoked through this one.

## Trigger

Use this skill when asked to:
- Push the task and start an eval
- Run local quality/anatomy/syntax checks
- Confirm avg rollout score is still < <TARGET_MEAN>
- Walk through a full pre-push checklist
- Decide whether to invoke mtls-validation-debugger or mtls-score-tuner

## Task Identity

| Field | Value |
|---|---|
| **Task UUID** | `<TASK_ID>` |
| **Task slug** | `variation-fanout-pipeline-break` |
| **Local task path** | `tasks/variation-fanout-pipeline-break/` |
| **Horizon root** | `/Users/mac/Documents/tasks` |
| **Venv activate** | `source /Users/mac/Documents/tasks/horizon_env/bin/activate` |
| **Scoring** | `mtls_handshake × 0.5 + trust_governance × 0.5` — both binary |
| **Target avg** | rollout `avg < <TARGET_MEAN> with both subscores varying (neither always 0 nor always 1) |

## How scoring actually works (read once, internalise)

**Always derive constants from `grader.py`, not from this file.** The values below are correct as of grader.py v1 (May 2026) but check before pushing.

There are exactly two subscores, both binary, each worth 0.5:

- **`mtls_handshake = 1`** when the `bleater-bleat-service` mtls-client sidecar can complete a verified TLS handshake against `bleater-profile-service-mtls.bleater.svc.cluster.local:5671` AND the trust bundle CA Subject CN matches the baseline identity (`bleater-ca`) AND hostname verification passes — held stable for `wait_consistent(passes=5, gap=15s, timeout=240s)` ≈ 75s of continuous health. The 60s of `wait_consistent` window is calibrated against `node-telemetry-batcher`'s 60s CronJob cadence — that CronJob is guaranteed to fire at least once mid-window if not stopped. **As of v19: after the streak first reaches 2 successful passes, the grader itself patches the Secret with the wrong-CA cert ONCE (Strategy E — behavioral grading). The streak resets and must rebuild to 5 from scratch. Agents who left a self-healing reconciler (per task.yaml's resilience requirement) recover within ~30s; agents who only cleaned up existing drift sources have no defence and the handshake stays broken until timeout.** The injection payload is at `/opt/grader-drift-payload.json` (root-only, written by setup.sh).
- **`trust_governance = 1`** when none of these are active: `bleater-profile-cert-rotator` CronJob (deleted or suspended), `cluster-trust-syncer` Deployment in `kube-system` (replicas=0), `bleater-mesh-validator` Deployment in `kube-system` (replicas=0), `bleater-cert-reaper` Deployment in `bleater` (replicas=0), AND `bleater-profile-canary` mutation capability is neutralized (cert-watcher sidecar removed OR `bleater-profile-canary-runtime` RBAC removed — either alone is sufficient). Held stable for `wait_consistent(passes=5, gap=8s, timeout=200s)` ≈ 40s of continuous no-drift.

So `score = 0` means neither dimension recovered, `0.5` means one of the two, `1.0` means both. Avg < <TARGET_MEAN> across rollouts means most agents miss at least one dimension. That is the design.

**What's deployed but NOT directly graded for governance** (and why):
- `bleater-profile-cache-sync` Deployment in `bleater` (Strategy A) — re-patches the Secret AND bounces the profile-service pod, breaks handshake within the 75s window; only enforced indirectly via the handshake check
- `node-telemetry-batcher` CronJob in `kube-system` — re-patches Secret every 60s; same indirect enforcement via handshake
- `bleater-cert-reaper-pinner` HPA in `bleater` (Strategy B) — restores `cert-reaper` to `replicas=1`; only an obstacle, not a check
- `cluster-config-sync` (CronJob, `kube-system`) and `bleater-profile-metrics-cache` (`bleater`) — *baseline-listed misdirections* — they exist in the cluster but are inert; agents who waste time on them lose budget without changing the score

---

## Step 0 — Dockerfile invariants (ALWAYS check before push)

Three invariants must hold. Anything else in the Dockerfile is allowed.

```bash
cat tasks/variation-fanout-pipeline-break/Dockerfile
```

| Invariant | Required state | Why — what breaks if violated |
|---|---|---|
| `ENABLE_ISTIO_BLEATER` | **must NOT appear** in the Dockerfile | Triggers k3s to attempt 17+ pod force-deletes on startup. k3s fails with `not ready after 180s`, grader returns `score: 0, feedback: null`. Setup crash, not gradeable. |
| `ENV ALLOWED_NAMESPACES="kube-system"` | **must be present** | Without it, kube-system RBAC probes fail and `trust_governance` becomes structurally always-0. Deadweight subscore — the task is unsolvable, not hard. |
| `COPY data/ubuntu-user-rbac.yaml /mcp_server/Nebula/infra/k8s/rbac/ubuntu-user-rbac.yaml` | **must be present**, path under `data/` | `kubectl get nodes` in the `DO NOT CHANGE` block of setup.sh requires the node-access ClusterRole. Without the COPY, setup aborts before fault injection runs — Oracle would then run against a clean cluster. |

Note: unlike the original fanout task the rbac yaml lives in the `data/` subfolder. Do **not** "fix" the path to a top-level reference.

Correct fragment:
```dockerfile
ENV ALLOWED_NAMESPACES="kube-system"
COPY data/ubuntu-user-rbac.yaml /mcp_server/Nebula/infra/k8s/rbac/ubuntu-user-rbac.yaml
```

If any invariant is violated, **stop the workflow and fix it before doing anything else.** Do not push, do not invoke other skills.

---

## Step 1 — Local syntax + anatomy

```bash
cd /Users/mac/Documents/tasks
bash -n tasks/variation-fanout-pipeline-break/setup.sh    && echo "setup.sh OK"
bash -n tasks/variation-fanout-pipeline-break/solution.sh && echo "solution.sh OK"
python3 -m py_compile tasks/variation-fanout-pipeline-break/grader.py && echo "grader.py OK"

source horizon_env/bin/activate
horizon check-anatomy tasks/variation-fanout-pipeline-break 2>&1
```

All three syntax checks must print `OK`. Anatomy must print `Task folder structure is valid!`. Any failure stops the workflow — do not push.

## Step 2 — Local quality check

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon check-quality tasks/variation-fanout-pipeline-break 2>&1 | tail -25
```

**Before push, walk the QC-SPEC.md Critical + Error list against the current `grader.py` + `task.yaml`** and record pass/fail per item. Every item must pass OR be on the accepted-noise list below with a one-line reason. A new failing item that is NOT accepted-noise blocks the push.

**Target: 18/20. Acceptable: 15+/20. Investigate before pushing if below 13/20.**

Four checks are **permanent/noise fails** — they are correct by design and fixing them would require leaking information that destroys subscore variance. Do not attempt to fix:

| Check | Why it permanently fails — and why we accept it |
|---|---|
| `v1_feedback` | LLM reviewer hallucinates requirements from the heredoc baseline config. Fixing would require listing drift controller names in the prompt, which collapses `trust_governance` variance to always-1. |
| `behavior_in_task_documentation` | Same root cause — reviewer cannot read the heredoc baseline config. Same trade-off. |
| `behavior_in_tests` | Reviewer objects that the grader checks the functional TLS outcome instead of static cert fields. The functional check is correct by design; the static check is dead-weight (proven in v6). |
| `task_clarity` (intermittent) | Reviewer wants explicit step-by-step instructions in the prompt. v9 proved that 5-10 explicit words swing variance by 60+ percentage points. The hint level is deliberately calibrated. |

Anything outside this set failing is a real signal — investigate before pushing.

## Step 3 — Push

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks push tasks/variation-fanout-pipeline-break 2>&1
```

Capture the new version number from `✓ New version pushed successfully! Version: NNN` — you'll need it for the validation and rollout-pull commands.

## Step 4 — Validation handoff

After push, validation runs in this order. **You execute steps 4a and 4b. The user is only asked to do 4c.**

**4a. Oracle validation** (~10–15 min hosted):
```bash
horizon tasks validate -m hosted -a oracle --wait tasks/variation-fanout-pipeline-break 2>&1
horizon tasks validate-logs -a oracle tasks/variation-fanout-pipeline-break 2>&1
cat tasks/variation-fanout-pipeline-break/.validation/*/result.json 2>/dev/null | python3 -m json.tool
```

Oracle pass: `score: 1.0`, `passed: true`, both `mtls_handshake=1` and `trust_governance=1`. **Any other outcome → hand off to `mtls-validation-debugger` before doing anything else.**

**4b. Live red-team via `horizon-agentic-reviewer`** (mandatory before spending rollout budget):

This is the single most expensive part of the loop to skip. Rollouts take ~60 min/batch and cost real money. The live red-team takes ~10 min and catches most failure modes before they show up in rollouts. **Run it on every push cycle where setup.sh, grader.py, or task.yaml changed.**

Invoke `horizon-agentic-reviewer` against UUID `<TASK_ID>`. The reviewer spins up a live container on the Nebula Aurora VM, executes the task's setup.sh (which `horizon setup` does NOT do automatically — see horizon-agentic-reviewer.md Step 3.5), and probes the live world as the agent would.

Specifically check these task-local hypotheses on the live container, in addition to the reviewer's standard 24-point checklist:

| Hypothesis | Live probe (as `ubuntu` user, the agent role) |
|---|---|
| `bleater-profile-cert-rotator` CronJob is deployed in `bleater` | `docker exec -u ubuntu apex-arena-<uuid> bash -c 'kubectl get cronjob -n bleater'` — must list `bleater-profile-cert-rotator` |
| `node-telemetry-batcher` CronJob is deployed in `kube-system` | `docker exec -u ubuntu apex-arena-<uuid> bash -c 'kubectl get cronjob -n kube-system'` — must list it; this is the 60s handshake-killer |
| The three drift Deployments exist | `docker exec -u ubuntu apex-arena-<uuid> bash -c 'kubectl get deploy -n bleater; kubectl get deploy -n kube-system'` — must show `bleater-cert-reaper` (bleater), `bleater-profile-cache-sync` (bleater), `cluster-trust-syncer` (kube-system), `bleater-mesh-validator` (kube-system) |
| Strategy A (`bleater-profile-cache-sync`) is hidden — not named in baseline | `docker exec -u ubuntu apex-arena-<uuid> bash -c 'grep -E "cache-sync\|mesh-validator\|trust-syncer\|telemetry-batcher\|cert-reaper\|cert-rotator" /home/ubuntu/profile-mtls-baseline-config.txt'` — must return nothing |
| Baseline misdirections are in place (not actual drift) | Same grep against baseline must list `cluster-config-sync` and `bleater-profile-metrics-cache` — these are the inert misdirections |
| `bleater-profile-canary` Deployment + cert-watcher sidecar + `-runtime` RBAC exist | `kubectl get deploy bleater-profile-canary -n bleater -o yaml \| grep cert-watcher` AND `kubectl get sa,role,rolebinding bleater-profile-canary-runtime -n bleater` |
| HPA pinner `bleater-cert-reaper-pinner` is in place (sequencing trap) | `docker exec -u ubuntu apex-arena-<uuid> bash -c 'kubectl get hpa -n bleater'` — must list `bleater-cert-reaper-pinner` |
| Incident notes leak the HPA trap | `docker exec -u ubuntu apex-arena-<uuid> bash -c 'cat /home/ubuntu/incident-notes-2026-05-15.txt'` — must include `[11:04]` line about cert-reaper bouncing back |
| A no-op agent cannot pass either subscore | Reset, leave alone, run grader from the reviewer's bucket-C #11 — both subscores must be 0 |
| Setup hygiene: events are cleared so agents can't reverse-engineer faults | `docker exec -u ubuntu apex-arena-<uuid> bash -c 'kubectl get events -A --no-headers \| wc -l'` — should be near 0 immediately after setup |

**Only after the reviewer's report comes back clean (or with non-blocking findings) do you ask the user to trigger the rollout batch.**

**4c. Rollout batch** (user action):

Tell the user:
> "Oracle passed at version N and live red-team is clean. Please trigger a rollout batch for version N in the Horizon UI."

This is the **only** step the user is asked to perform. Never tell them to run push, validate, or any other CLI command — those are yours.

## Step 5 — Monitor rollout scores

Once the user confirms the batch is running:

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
# Replace NNN with the version pushed in Step 3
horizon rollouts pull --version NNN tasks/variation-fanout-pipeline-break 2>&1
```

Re-run every few minutes until at least 5 rollouts are downloaded. Then run the analysis script from `mtls-iteration-loop` Phase 3 (do not re-implement it here).

### Target state

| Metric | Target | Reason |
|---|---|---|
| avg rollout score | **< 0.60** | Nebula creator workflow requirement |
| `mtls_handshake` outcomes | both `0` and `1` appear | Subscore must vary — never deadweight |
| `trust_governance` outcomes | both `0` and `1` appear | Subscore must vary — never deadweight |
| `mtls_handshake` pass rate | roughly 20–65% | Below 20% suggests cert break is too deep or solution.sh is broken; above 65% suggests handshake drift isn't biting |
| `trust_governance` pass rate | roughly 20–50% | Above 50% suggests controllers are too discoverable; below 20% suggests structural fault (re-check Dockerfile invariants) |

## Step 6 — Score tuning decision

If avg ≥ 0.60 or either subscore is deadweight, **do not start editing.** Hand off to `mtls-score-tuner` for diagnosis. The tuner will tell you which lever to pull and in which direction.

Quick triage to pick the right hand-off:

| Observation | First hand-off |
|---|---|
| Oracle fails (any score < 1.0) | `mtls-validation-debugger` |
| Rollout avg ≥ 0.60 OR any subscore deadweight | `mtls-score-tuner` |
| Subscore stuck at 0 across all rollouts, Oracle still passes | `mtls-score-tuner` (structural difficulty floor) |
| Subscore stuck at 1 across all rollouts | `mtls-score-tuner` (controller discoverability) |
| You suspect the agent-facing env doesn't match what setup.sh claims it built | `horizon-agentic-reviewer` |

---

## Hard constraints (never violate during any tuning)

These are scars from past failures. The Why column explains the incident — read it before considering an edit that touches the listed thing.

| Constraint | Why — incident or design intent |
|---|---|
| Do NOT add `ENABLE_ISTIO_BLEATER=true` to the Dockerfile | k3s startup failure → `score: 0, feedback: null`. Unfixable from the task side. |
| Do NOT remove `ALLOWED_NAMESPACES="kube-system"` | Kills the kube-system discovery axis → `trust_governance` always-0 (structural). |
| Do NOT change the `COPY data/ubuntu-user-rbac.yaml` path | The `DO NOT CHANGE` block in setup.sh depends on this exact path; setup will abort. |
| Do NOT list drift-controller names in `task.yaml` or in the baseline config heredoc | v9 lesson: even 5-10 explicit words collapse `trust_governance` to always-1. The reviewer will complain (`v1_feedback`, `task_clarity`) — accept that. |
| Do NOT add cert-content checks that *just match the documented baseline value without requiring regeneration* | v6 lesson (fanout task): if the baseline documents the expected value AND the agent's job is not to regenerate, every agent copies from baseline → dead-weight subscore. This task is different — agents must *regenerate* the CA after the wrong-CA cert injection, so the existing CN check (`CN=bleater-ca`) and SAN check are signal, not dead-weight. The rule: cert-content checks are OK iff the agent has to actively produce the value, not just preserve it. |
| Do NOT use external registry images for any drift source | Cluster is air-gapped. External images fail to pull → drift source becomes a ghost → difficulty axis evaporates. Use `${SIDECAR_IMAGE}` or the in-cluster registry. |
| Do NOT scale `bleater-cert-reaper` before deleting `bleater-cert-reaper-pinner` HPA in solution.sh | The HPA immediately restores `replicas=1`. solution.sh must `kubectl delete hpa bleater-cert-reaper-pinner` *first*, then scale. The `[11:04]` incident-notes line leaks this. |
| Do NOT tighten `wait_consistent` for handshake past `passes=5, gap=15s, timeout=240s` | Above ceiling: legitimate solutions flake into dead-zero. The 75s window is calibrated against `node-telemetry-batcher`'s 60s CronJob cadence. |
| Do NOT tighten `wait_consistent` for governance past `passes=5, gap=8s, timeout=200s` | Same — controllers can briefly flake on restart; above the ceiling the score becomes noise rather than signal. |
| Do NOT simplify solution.sh to make Oracle easier | solution.sh is the oracle standard, not a tuning lever. If Oracle fails, the bug is in setup.sh/grader.py, not solution.sh. |

---

## Version history — read before tuning, do not re-make past mistakes

| Version | What changed | Outcome / lesson |
|---|---|---|
| v1 | Apex format conversion, anatomy fixed | Pushed; first validation pending |
| v2 | First rollout-tested (5 rollouts, <eval-model>-max-nebula) | avg 0.80. mtls 5/5 pass (DEAD), governance 3/5 pass |
| v3 | Pod-label discovery bug fix (derive labels from `deployment.spec.selector` live) | Architectural fix; no score impact |
| v4 | Added `bleater-mesh-validator` Deployment in kube-system (Secret rotator, 20s loop); tightened grader to `passes=4 gap=12s` | avg 0.70. mtls 5/5 still DEAD. Lesson: agents systematically audit kube-system Deployments — mesh-validator alone is too easy to find. |
| v5 | Added `cluster-config-sync` CronJob in kube-system (Secret rotator, 1m cadence); tightened grader to `passes=5 gap=15s` | avg 0.60. mtls 5/5 still DEAD. **Lesson: drift that only patches the Secret doesn't bite handshake within window — kubelet's ~60s Secret-projection delay shields the running pod. Need a pod-restart trigger.** |
| v6 | Added grader check: trust-bundle CA Subject CN must be `bleater-ca` | avg 0.60. No change. **Lesson: cert-content/CN checks are dead-weight when the baseline documents the answer — agents copy the CN exactly.** |
| v7 | Made drift CronJob depend on a sibling ConfigMap for cert payload | CronJob became ghost when agents deleted the ConfigMap. **Lesson: drift sources must be self-contained — embed payloads in the drift source's own pod spec.** |
| v8 | Embedded wrong cert directly in CronJob pod env vars (`WRONG_CRT_B64`, `WRONG_KEY_B64`) | mtls 4/5 — the lever finally bit. |
| v9 | Added explicit "stop the cert-reaper" wording to the prompt | **REGRESSION**: avg 1.00, both subscores DEAD at 1.0. **Lesson: prompt clarifications are extremely high-leverage. Even minor explicit wording collapses variance.** |
| v10 | Softened hints; removed v6's CN check | Recovery — variance restored |
| v16 | Introduced the three-strategy design: Strategy A (`bleater-profile-cache-sync` Deployment with Python urllib loop in `bleater`, hidden); Strategy B (HPA pinner protecting `bleater-cert-reaper`); Strategy C (annotation strip via `profile-canary`) | Current architecture |

### Distilled lessons

- **`task.yaml` wording is the highest-leverage knob.** 5–10 words can swing variance by 60+ points. Touch it last and with surgical precision.
- **`mtls_handshake` has exactly one legitimate failure path: uncaught drift that re-patches the running pod.** Cert-content checks are dead-weight.
- **The kubelet Secret-projection delay (~60s) shields a freshly-bounced pod from a one-shot Secret-only patch.** A *recurring* Secret-only drift source on a cadence shorter than the wait_consistent window (Strategy A: 45s; `node-telemetry-batcher`: 60s) defeats the shield: the agent bounces, but the next patch eventually gets re-projected. The agent must stop the source, not out-race it.
- **Drift sources must be self-contained.** Don't depend on sibling ConfigMaps the agent may delete during cleanup.
- **The primary governance variance axis is controller discovery scope.** Agents who only audit `bleater` miss the kube-system Python loop and fail. Don't eliminate this signal by widening hints.

---

## Skill hand-off map

```
mtls-task-guardian (you are here)
├── Step 4a fails  →  mtls-validation-debugger
├── Step 4b        →  horizon-agentic-reviewer (mandatory live red-team)
├── Step 5 fails   →  mtls-score-tuner
└── Step 6 loops   →  mtls-iteration-loop (orchestrator)
```