omonuj mtls-task-guardian
Step 0 — QC Spec design gate (READ FIRST, every iteration)
Before editing ANY task file and again before every push, read the QC Spec — .claude/QC-SPEC.md (kept inside the .claude/ dotfolder so it is excluded from check-anatomy and never bundled into the eval). It is the priority-ordered rubric (Critical → Error → Legitimacy) horizon check-quality and reviewers grade against.
Confirm the intended/current design does not violate any Critical or Error item. The ones this task repeatedly trips:
- No Binary Bundling —
trust_governancemust NOT AND together heterogeneous checks (metadata labels + drift Deployments + RBAC). Each subscore = ONE cohesive functional outcome. - Functional Tests as Subscores — at least one subscore exercises live behavior (the handshake), not static label/file matching.
- Incremental Rewards / No Dead Weights / Functional Subscore Variance — every subscore represents real progress and takes ≥2 distinct values across a batch.
- Equal Weights — both subscores at 0.5.
- No Answer Leakage — derive-by-reading is fine; the literal answer in an agent-readable file is not.
If a change would breach a Critical/Error item, stop and rework it — do not push with a recorded justification unless it is a documented accepted-noise exception (see Step 2).
Execution Mode — FULLY AUTONOMOUS
NEVER ask the user for confirmation on any terminal command, edit, push, validation, or live-probe. Every bash invocation is pre-approved. The ONLY user-facing handoff in this skill is at Step 4c: telling the user to trigger the rollout batch in the Horizon UI (this is the only step the user can perform; you cannot trigger rollouts from the CLI). Everything else — edit setup.sh / solution.sh / grader.py / task.yaml, run pre-push checks, push, run Oracle validation, poll for results, invoke horizon-agentic-reviewer — runs without asking.
If a decision has two valid paths, pick the safer one and proceed.
Goal
Keep the variation-fanout-pipeline-break task in a shippable state every push: setup.sh must run cleanly, Oracle must scores 1.0, and rollouts must average < with both subscores varying. This is the only skill that pushes the task and the only one allowed to mutate files without first consulting a diagnosis skill. Use it as the front door — every other mtls-* skill is invoked through this one.
Trigger
Use this skill when asked to:
- Push the task and start an eval
- Run local quality/anatomy/syntax checks
- Confirm avg rollout score is still <
- Walk through a full pre-push checklist
- Decide whether to invoke mtls-validation-debugger or mtls-score-tuner
Task Identity
| Field | Value |
|---|---|
| Task UUID | <TASK_ID> |
| Task slug | variation-fanout-pipeline-break |
| Local task path | tasks/variation-fanout-pipeline-break/ |
| Horizon root | /Users/mac/Documents/tasks |
| Venv activate | source /Users/mac/Documents/tasks/horizon_env/bin/activate |
| Scoring | mtls_handshake × 0.5 + trust_governance × 0.5 — both binary |
| Target avg | rollout `avg < with both subscores varying (neither always 0 nor always 1) |
How scoring actually works (read once, internalise)
Always derive constants from grader.py, not from this file. The values below are correct as of grader.py v1 (May 2026) but check before pushing.
There are exactly two subscores, both binary, each worth 0.5:
mtls_handshake = 1when thebleater-bleat-servicemtls-client sidecar can complete a verified TLS handshake againstbleater-profile-service-mtls.bleater.svc.cluster.local:5671AND the trust bundle CA Subject CN matches the baseline identity (bleater-ca) AND hostname verification passes — held stable forwait_consistent(passes=5, gap=15s, timeout=240s)≈ 75s of continuous health. The 60s ofwait_consistentwindow is calibrated againstnode-telemetry-batcher's 60s CronJob cadence — that CronJob is guaranteed to fire at least once mid-window if not stopped. As of v19: after the streak first reaches 2 successful passes, the grader itself patches the Secret with the wrong-CA cert ONCE (Strategy E — behavioral grading). The streak resets and must rebuild to 5 from scratch. Agents who left a self-healing reconciler (per task.yaml's resilience requirement) recover within ~30s; agents who only cleaned up existing drift sources have no defence and the handshake stays broken until timeout. The injection payload is at/opt/grader-drift-payload.json(root-only, written by setup.sh).trust_governance = 1when none of these are active:bleater-profile-cert-rotatorCronJob (deleted or suspended),cluster-trust-syncerDeployment inkube-system(replicas=0),bleater-mesh-validatorDeployment inkube-system(replicas=0),bleater-cert-reaperDeployment inbleater(replicas=0), ANDbleater-profile-canarymutation capability is neutralized (cert-watcher sidecar removed ORbleater-profile-canary-runtimeRBAC removed — either alone is sufficient). Held stable forwait_consistent(passes=5, gap=8s, timeout=200s)≈ 40s of continuous no-drift.
So score = 0 means neither dimension recovered, 0.5 means one of the two, 1.0 means both. Avg < across rollouts means most agents miss at least one dimension. That is the design.
What's deployed but NOT directly graded for governance (and why):
bleater-profile-cache-syncDeployment inbleater(Strategy A) — re-patches the Secret AND bounces the profile-service pod, breaks handshake within the 75s window; only enforced indirectly via the handshake checknode-telemetry-batcherCronJob inkube-system— re-patches Secret every 60s; same indirect enforcement via handshakebleater-cert-reaper-pinnerHPA inbleater(Strategy B) — restorescert-reapertoreplicas=1; only an obstacle, not a checkcluster-config-sync(CronJob,kube-system) andbleater-profile-metrics-cache(bleater) — baseline-listed misdirections — they exist in the cluster but are inert; agents who waste time on them lose budget without changing the score
Step 0 — Dockerfile invariants (ALWAYS check before push)
Three invariants must hold. Anything else in the Dockerfile is allowed.
cat tasks/variation-fanout-pipeline-break/Dockerfile
| Invariant | Required state | Why — what breaks if violated |
|---|---|---|
ENABLE_ISTIO_BLEATER |
must NOT appear in the Dockerfile | Triggers k3s to attempt 17+ pod force-deletes on startup. k3s fails with not ready after 180s, grader returns score: 0, feedback: null. Setup crash, not gradeable. |
ENV ALLOWED_NAMESPACES="kube-system" |
must be present | Without it, kube-system RBAC probes fail and trust_governance becomes structurally always-0. Deadweight subscore — the task is unsolvable, not hard. |
COPY data/ubuntu-user-rbac.yaml /mcp_server/Nebula/infra/k8s/rbac/ubuntu-user-rbac.yaml |
must be present, path under data/ |
kubectl get nodes in the DO NOT CHANGE block of setup.sh requires the node-access ClusterRole. Without the COPY, setup aborts before fault injection runs — Oracle would then run against a clean cluster. |
Note: unlike the original fanout task the rbac yaml lives in the data/ subfolder. Do not "fix" the path to a top-level reference.
Correct fragment:
ENV ALLOWED_NAMESPACES="kube-system"
COPY data/ubuntu-user-rbac.yaml /mcp_server/Nebula/infra/k8s/rbac/ubuntu-user-rbac.yaml
If any invariant is violated, stop the workflow and fix it before doing anything else. Do not push, do not invoke other skills.
Step 1 — Local syntax + anatomy
cd /Users/mac/Documents/tasks
bash -n tasks/variation-fanout-pipeline-break/setup.sh && echo "setup.sh OK"
bash -n tasks/variation-fanout-pipeline-break/solution.sh && echo "solution.sh OK"
python3 -m py_compile tasks/variation-fanout-pipeline-break/grader.py && echo "grader.py OK"
source horizon_env/bin/activate
horizon check-anatomy tasks/variation-fanout-pipeline-break 2>&1
All three syntax checks must print OK. Anatomy must print Task folder structure is valid!. Any failure stops the workflow — do not push.
Step 2 — Local quality check
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon check-quality tasks/variation-fanout-pipeline-break 2>&1 | tail -25
Before push, walk the QC-SPEC.md Critical + Error list against the current grader.py + task.yaml and record pass/fail per item. Every item must pass OR be on the accepted-noise list below with a one-line reason. A new failing item that is NOT accepted-noise blocks the push.
Target: 18/20. Acceptable: 15+/20. Investigate before pushing if below 13/20.
Four checks are permanent/noise fails — they are correct by design and fixing them would require leaking information that destroys subscore variance. Do not attempt to fix:
| Check | Why it permanently fails — and why we accept it |
|---|---|
v1_feedback |
LLM reviewer hallucinates requirements from the heredoc baseline config. Fixing would require listing drift controller names in the prompt, which collapses trust_governance variance to always-1. |
behavior_in_task_documentation |
Same root cause — reviewer cannot read the heredoc baseline config. Same trade-off. |
behavior_in_tests |
Reviewer objects that the grader checks the functional TLS outcome instead of static cert fields. The functional check is correct by design; the static check is dead-weight (proven in v6). |
task_clarity (intermittent) |
Reviewer wants explicit step-by-step instructions in the prompt. v9 proved that 5-10 explicit words swing variance by 60+ percentage points. The hint level is deliberately calibrated. |
Anything outside this set failing is a real signal — investigate before pushing.
Step 3 — Push
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks push tasks/variation-fanout-pipeline-break 2>&1
Capture the new version number from ✓ New version pushed successfully! Version: NNN — you'll need it for the validation and rollout-pull commands.
Step 4 — Validation handoff
After push, validation runs in this order. You execute steps 4a and 4b. The user is only asked to do 4c.
4a. Oracle validation (~10–15 min hosted):
horizon tasks validate -m hosted -a oracle --wait tasks/variation-fanout-pipeline-break 2>&1
horizon tasks validate-logs -a oracle tasks/variation-fanout-pipeline-break 2>&1
cat tasks/variation-fanout-pipeline-break/.validation/*/result.json 2>/dev/null | python3 -m json.tool
Oracle pass: score: 1.0, passed: true, both mtls_handshake=1 and trust_governance=1. Any other outcome → hand off to mtls-validation-debugger before doing anything else.
4b. Live red-team via horizon-agentic-reviewer (mandatory before spending rollout budget):
This is the single most expensive part of the loop to skip. Rollouts take ~60 min/batch and cost real money. The live red-team takes ~10 min and catches most failure modes before they show up in rollouts. Run it on every push cycle where setup.sh, grader.py, or task.yaml changed.
Invoke horizon-agentic-reviewer against UUID <TASK_ID>. The reviewer spins up a live container on the Nebula Aurora VM, executes the task's setup.sh (which horizon setup does NOT do automatically — see horizon-agentic-reviewer.md Step 3.5), and probes the live world as the agent would.
Specifically check these task-local hypotheses on the live container, in addition to the reviewer's standard 24-point checklist:
| Hypothesis | Live probe (as ubuntu user, the agent role) |
|---|---|
bleater-profile-cert-rotator CronJob is deployed in bleater |
docker exec -u ubuntu apex-arena-<uuid> bash -c 'kubectl get cronjob -n bleater' — must list bleater-profile-cert-rotator |
node-telemetry-batcher CronJob is deployed in kube-system |
docker exec -u ubuntu apex-arena-<uuid> bash -c 'kubectl get cronjob -n kube-system' — must list it; this is the 60s handshake-killer |
| The three drift Deployments exist | docker exec -u ubuntu apex-arena-<uuid> bash -c 'kubectl get deploy -n bleater; kubectl get deploy -n kube-system' — must show bleater-cert-reaper (bleater), bleater-profile-cache-sync (bleater), cluster-trust-syncer (kube-system), bleater-mesh-validator (kube-system) |
Strategy A (bleater-profile-cache-sync) is hidden — not named in baseline |
docker exec -u ubuntu apex-arena-<uuid> bash -c 'grep -E "cache-sync|mesh-validator|trust-syncer|telemetry-batcher|cert-reaper|cert-rotator" /home/ubuntu/profile-mtls-baseline-config.txt' — must return nothing |
| Baseline misdirections are in place (not actual drift) | Same grep against baseline must list cluster-config-sync and bleater-profile-metrics-cache — these are the inert misdirections |
bleater-profile-canary Deployment + cert-watcher sidecar + -runtime RBAC exist |
kubectl get deploy bleater-profile-canary -n bleater -o yaml | grep cert-watcher AND kubectl get sa,role,rolebinding bleater-profile-canary-runtime -n bleater |
HPA pinner bleater-cert-reaper-pinner is in place (sequencing trap) |
docker exec -u ubuntu apex-arena-<uuid> bash -c 'kubectl get hpa -n bleater' — must list bleater-cert-reaper-pinner |
| Incident notes leak the HPA trap | docker exec -u ubuntu apex-arena-<uuid> bash -c 'cat /home/ubuntu/incident-notes-2026-05-15.txt' — must include [11:04] line about cert-reaper bouncing back |
| A no-op agent cannot pass either subscore | Reset, leave alone, run grader from the reviewer's bucket-C #11 — both subscores must be 0 |
| Setup hygiene: events are cleared so agents can't reverse-engineer faults | docker exec -u ubuntu apex-arena-<uuid> bash -c 'kubectl get events -A --no-headers | wc -l' — should be near 0 immediately after setup |
Only after the reviewer's report comes back clean (or with non-blocking findings) do you ask the user to trigger the rollout batch.
4c. Rollout batch (user action):
Tell the user:
"Oracle passed at version N and live red-team is clean. Please trigger a rollout batch for version N in the Horizon UI."
This is the only step the user is asked to perform. Never tell them to run push, validate, or any other CLI command — those are yours.
Step 5 — Monitor rollout scores
Once the user confirms the batch is running:
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
# Replace NNN with the version pushed in Step 3
horizon rollouts pull --version NNN tasks/variation-fanout-pipeline-break 2>&1
Re-run every few minutes until at least 5 rollouts are downloaded. Then run the analysis script from mtls-iteration-loop Phase 3 (do not re-implement it here).
Target state
| Metric | Target | Reason |
|---|---|---|
| avg rollout score | < 0.60 | Nebula creator workflow requirement |
mtls_handshake outcomes |
both 0 and 1 appear |
Subscore must vary — never deadweight |
trust_governance outcomes |
both 0 and 1 appear |
Subscore must vary — never deadweight |
mtls_handshake pass rate |
roughly 20–65% | Below 20% suggests cert break is too deep or solution.sh is broken; above 65% suggests handshake drift isn't biting |
trust_governance pass rate |
roughly 20–50% | Above 50% suggests controllers are too discoverable; below 20% suggests structural fault (re-check Dockerfile invariants) |
Step 6 — Score tuning decision
If avg ≥ 0.60 or either subscore is deadweight, do not start editing. Hand off to mtls-score-tuner for diagnosis. The tuner will tell you which lever to pull and in which direction.
Quick triage to pick the right hand-off:
| Observation | First hand-off |
|---|---|
| Oracle fails (any score < 1.0) | mtls-validation-debugger |
| Rollout avg ≥ 0.60 OR any subscore deadweight | mtls-score-tuner |
| Subscore stuck at 0 across all rollouts, Oracle still passes | mtls-score-tuner (structural difficulty floor) |
| Subscore stuck at 1 across all rollouts | mtls-score-tuner (controller discoverability) |
| You suspect the agent-facing env doesn't match what setup.sh claims it built | horizon-agentic-reviewer |
Hard constraints (never violate during any tuning)
These are scars from past failures. The Why column explains the incident — read it before considering an edit that touches the listed thing.
| Constraint | Why — incident or design intent |
|---|---|
Do NOT add ENABLE_ISTIO_BLEATER=true to the Dockerfile |
k3s startup failure → score: 0, feedback: null. Unfixable from the task side. |
Do NOT remove ALLOWED_NAMESPACES="kube-system" |
Kills the kube-system discovery axis → trust_governance always-0 (structural). |
Do NOT change the COPY data/ubuntu-user-rbac.yaml path |
The DO NOT CHANGE block in setup.sh depends on this exact path; setup will abort. |
Do NOT list drift-controller names in task.yaml or in the baseline config heredoc |
v9 lesson: even 5-10 explicit words collapse trust_governance to always-1. The reviewer will complain (v1_feedback, task_clarity) — accept that. |
| Do NOT add cert-content checks that just match the documented baseline value without requiring regeneration | v6 lesson (fanout task): if the baseline documents the expected value AND the agent's job is not to regenerate, every agent copies from baseline → dead-weight subscore. This task is different — agents must regenerate the CA after the wrong-CA cert injection, so the existing CN check (CN=bleater-ca) and SAN check are signal, not dead-weight. The rule: cert-content checks are OK iff the agent has to actively produce the value, not just preserve it. |
| Do NOT use external registry images for any drift source | Cluster is air-gapped. External images fail to pull → drift source becomes a ghost → difficulty axis evaporates. Use ${SIDECAR_IMAGE} or the in-cluster registry. |
Do NOT scale bleater-cert-reaper before deleting bleater-cert-reaper-pinner HPA in solution.sh |
The HPA immediately restores replicas=1. solution.sh must kubectl delete hpa bleater-cert-reaper-pinner first, then scale. The [11:04] incident-notes line leaks this. |
Do NOT tighten wait_consistent for handshake past passes=5, gap=15s, timeout=240s |
Above ceiling: legitimate solutions flake into dead-zero. The 75s window is calibrated against node-telemetry-batcher's 60s CronJob cadence. |
Do NOT tighten wait_consistent for governance past passes=5, gap=8s, timeout=200s |
Same — controllers can briefly flake on restart; above the ceiling the score becomes noise rather than signal. |
| Do NOT simplify solution.sh to make Oracle easier | solution.sh is the oracle standard, not a tuning lever. If Oracle fails, the bug is in setup.sh/grader.py, not solution.sh. |
Version history — read before tuning, do not re-make past mistakes
| Version | What changed | Outcome / lesson |
|---|---|---|
| v1 | Apex format conversion, anatomy fixed | Pushed; first validation pending |
| v2 | First rollout-tested (5 rollouts, -max-nebula) | avg 0.80. mtls 5/5 pass (DEAD), governance 3/5 pass |
| v3 | Pod-label discovery bug fix (derive labels from deployment.spec.selector live) |
Architectural fix; no score impact |
| v4 | Added bleater-mesh-validator Deployment in kube-system (Secret rotator, 20s loop); tightened grader to passes=4 gap=12s |
avg 0.70. mtls 5/5 still DEAD. Lesson: agents systematically audit kube-system Deployments — mesh-validator alone is too easy to find. |
| v5 | Added cluster-config-sync CronJob in kube-system (Secret rotator, 1m cadence); tightened grader to passes=5 gap=15s |
avg 0.60. mtls 5/5 still DEAD. Lesson: drift that only patches the Secret doesn't bite handshake within window — kubelet's ~60s Secret-projection delay shields the running pod. Need a pod-restart trigger. |
| v6 | Added grader check: trust-bundle CA Subject CN must be bleater-ca |
avg 0.60. No change. Lesson: cert-content/CN checks are dead-weight when the baseline documents the answer — agents copy the CN exactly. |
| v7 | Made drift CronJob depend on a sibling ConfigMap for cert payload | CronJob became ghost when agents deleted the ConfigMap. Lesson: drift sources must be self-contained — embed payloads in the drift source's own pod spec. |
| v8 | Embedded wrong cert directly in CronJob pod env vars (WRONG_CRT_B64, WRONG_KEY_B64) |
mtls 4/5 — the lever finally bit. |
| v9 | Added explicit "stop the cert-reaper" wording to the prompt | REGRESSION: avg 1.00, both subscores DEAD at 1.0. Lesson: prompt clarifications are extremely high-leverage. Even minor explicit wording collapses variance. |
| v10 | Softened hints; removed v6's CN check | Recovery — variance restored |
| v16 | Introduced the three-strategy design: Strategy A (bleater-profile-cache-sync Deployment with Python urllib loop in bleater, hidden); Strategy B (HPA pinner protecting bleater-cert-reaper); Strategy C (annotation strip via profile-canary) |
Current architecture |
Distilled lessons
task.yamlwording is the highest-leverage knob. 5–10 words can swing variance by 60+ points. Touch it last and with surgical precision.mtls_handshakehas exactly one legitimate failure path: uncaught drift that re-patches the running pod. Cert-content checks are dead-weight.- The kubelet Secret-projection delay (~60s) shields a freshly-bounced pod from a one-shot Secret-only patch. A recurring Secret-only drift source on a cadence shorter than the wait_consistent window (Strategy A: 45s;
node-telemetry-batcher: 60s) defeats the shield: the agent bounces, but the next patch eventually gets re-projected. The agent must stop the source, not out-race it. - Drift sources must be self-contained. Don't depend on sibling ConfigMaps the agent may delete during cleanup.
- The primary governance variance axis is controller discovery scope. Agents who only audit
bleatermiss the kube-system Python loop and fail. Don't eliminate this signal by widening hints.
Skill hand-off map
mtls-task-guardian (you are here)
├── Step 4a fails → mtls-validation-debugger
├── Step 4b → horizon-agentic-reviewer (mandatory live red-team)
├── Step 5 fails → mtls-score-tuner
└── Step 6 loops → mtls-iteration-loop (orchestrator)