# Mtls Validation Debugger

> Fetches and interprets validation logs (No-Op and Oracle) for the variation-fanout-pipeline-break task (<TASK_ID>). Maps observed log patterns to root causes and the smallest safe fix. Use after a validation fails to understand why it failed before editing anything.

- Skill: `omonuj/mtls-validation-debugger` (Agent Skill)
- Install (CLI): `npx skillmds@latest add omonuj/mtls-validation-debugger`
- Raw SKILL.md: https://api.skillmd.com/api/skills/omonuj/mtls-validation-debugger/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: omonuj (https://skillmd.com/u/omonuj)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/omonuj/mtls-validation-debugger

---


# mtls-validation-debugger

## Execution Mode — FULLY AUTONOMOUS

**NEVER ask the user for confirmation.** Fetch logs, interpret results, propose and apply fixes immediately. The only user-facing handoff in the loop is triggering rollouts in the Horizon UI (owned by `mtls-iteration-loop` Step 4c). This skill never has user-facing handoffs.

## Goal

Turn an Oracle or No-Op failure into a specific root cause with a specific fix, using only the log evidence — not guesswork. The point of this skill is to prevent blind edits. If you cannot map the observed log pattern to one of the rows below with confidence, escalate to `horizon-agentic-reviewer` for a live probe before changing any file.

## Trigger

Use this skill when:
- Oracle returned anything other than `score: 1.0, passed: true`
- No-Op returned a score > 0 (false positive — the grader passed on a broken setup)
- The grader crashed with a Python exception
- `result.json` shows `feedback: null` (container/setup crash before grader ran)
- You are about to edit setup.sh, solution.sh, or grader.py in response to a validation failure

Do **not** use this skill for rollout failures (those are scoring/variance issues — use `mtls-score-tuner`). Oracle and No-Op are validation; rollouts are evaluation.

## Task Identity

| Field | Value |
|---|---|
| **Task UUID** | `<TASK_ID>` |
| **Task slug** | `variation-fanout-pipeline-break` |
| **Local path** | `tasks/variation-fanout-pipeline-break/` |
| **Subscores** | `mtls_handshake` (0.5) + `trust_governance` (0.5), both binary |
| **Horizon root** | `/Users/mac/Documents/tasks` |
| **Venv** | `source /Users/mac/Documents/tasks/horizon_env/bin/activate` |

## How to read a validation result (read once, internalise)

Oracle runs `setup.sh → solution.sh → grader.py`. No-Op runs `setup.sh → grader.py` (no solution). The four possible result shapes:

| `score` | `passed` | `feedback` | Meaning | Where to look |
|---|---|---|---|---|
| `0` | `false` | `null` | Container or setup crashed before the grader could run | `output.txt` for the crash trace |
| `0` | `false` | non-null string | Grader ran end-to-end, both subscores failed | grader feedback for the specific check that fired |
| `0.5` | `false` | non-null string | One subscore passed, the other failed | feedback names which subscore is 1 vs 0 |
| `1.0` | `true` | non-null string | Both subscores passed | nothing to debug |

For No-Op, **anything other than `score: 0` is a problem** — the grader should never pass without solution.sh having run. A No-Op pass means the grader is too lax or setup.sh didn't actually break the world.

---

## Step 1 — Fetch the logs

```bash
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate

# Oracle (setup + solution + grader)
horizon tasks validate-logs -a oracle tasks/variation-fanout-pipeline-break 2>&1

# No-Op (setup + grader, no solution) — only when iteration-loop says to run it
horizon tasks validate-logs -a noop tasks/variation-fanout-pipeline-break 2>&1
```

Logs persist under:
- `tasks/variation-fanout-pipeline-break/.validation/<build_id>/output.txt`
- `tasks/variation-fanout-pipeline-break/.validation/<build_id>/result.json`

## Step 2 — Read `result.json` first

```bash
cat tasks/variation-fanout-pipeline-break/.validation/*/result.json 2>/dev/null | python3 -m json.tool
```

The four-row table above tells you which symptom branch to follow in Step 3. **Do not skip to `output.txt` without checking `result.json` first** — the result shape narrows the search dramatically.

## Step 3 — Map symptom → cause → fix

### Branch A. `feedback: null` / `score: 0` — setup crash

Setup never finished, so the grader never ran. Search `output.txt` for the first error line:

| Pattern in `output.txt` | Root cause | Fix |
|---|---|---|
| `k3s is not ready after 180 seconds` | `ENABLE_ISTIO_BLEATER=true` is in the Dockerfile, causing 17+ pod force-deletes on startup | Remove `ENABLE_ISTIO_BLEATER` from the Dockerfile entirely |
| `Error from server (Forbidden)` on `kubectl get nodes` | `data/ubuntu-user-rbac.yaml` not copied into image, or COPY path wrong | Restore `COPY data/ubuntu-user-rbac.yaml /mcp_server/Nebula/infra/k8s/rbac/ubuntu-user-rbac.yaml` |
| `Error from server (Forbidden)` on kube-system resources | `ALLOWED_NAMESPACES` env var missing | Add `ENV ALLOWED_NAMESPACES="kube-system"` to Dockerfile |
| `failed to pull image ... bleater-profile-cache-sync` | Strategy A deployment is referencing an external registry image | Use `${SIDECAR_IMAGE}` (the in-cluster reference); cluster is air-gapped |
| `Error: failed to pull image` (any other image) | Base image or tag wrong | Verify `FROM` line in Dockerfile |
| `kubectl: command not found` | Wrong base image | Verify `FROM` line in Dockerfile |
| `setup.sh: line N: syntax error` | Bash syntax error | Run `bash -n setup.sh` locally — fix that line |
| Setup stalls / hits hosted timeout with no error | k3s init slow OR a `kubectl wait` is hanging | First verify `ENABLE_ISTIO_BLEATER` absent; then add `--timeout=` flags to any unbounded `kubectl wait` |

### Branch B. No-Op returned `score > 0` — false positive

The grader passed without solution.sh having run. Either setup.sh did not break the world, or the grader is too lax.

| Subscore that passed on No-Op | Why it shouldn't have | Fix |
|---|---|---|
| `mtls_handshake = 1` on No-Op | Drift didn't bite — either `bleater-profile-cache-sync` or `node-telemetry-batcher` failed to deploy, OR the wait_consistent window is shorter than both their cadences | `kubectl get deploy bleater-profile-cache-sync -n bleater` (must exist, replicas=1) AND `kubectl get cronjob node-telemetry-batcher -n kube-system` (must exist, not suspended); also confirm grader.py `wait_consistent` window is 240s (handshake) — anything shorter than ~120s lets a lucky agent slip through |
| `trust_governance = 1` on No-Op | One or more drift controllers didn't deploy. Check setup.sh fault-injection section is reaching all four targets | `grep -E "cert-reaper\|cache-sync\|canary\|hpa-pinner\|kube-system.*python" /tmp/noop_log.txt` — every name must appear in the setup output |

If setup.sh appears to claim it deployed everything but the cluster is missing pieces, add `set -ex` to setup.sh temporarily, push, and re-run No-Op to get verbose trace. **The fault-injection section often fails silently on `|| true` paths.**

### Branch C. Oracle `score: 0.5` — `mtls_handshake` failed (governance passed)

Solution.sh stopped the drift controllers but the TLS handshake never recovered. Search for handshake-relevant log lines:

```bash
grep -i "handshake\|cert\|ssl\|tls\|verify" /tmp/oracle_log.txt | head -30
```

Common causes, ordered by likelihood:

| Cause | Why | Fix in solution.sh |
|---|---|---|
| Pods not bounced after Secret patch | New cert is on disk but the running process holds the old one; kubelet's ~60s Secret-projection delay keeps the new mount hidden | Add `kubectl delete pod -l app=<profile-service-pod-label> -n bleater --wait=false` AND `kubectl delete pod -l app=<bleat-service-pod-label> -n bleater --wait=false` after the Secret patch. solution.sh lines 201-202 do this. |
| Wrong CA Subject CN on regenerated CA | grader.py line 174 checks `cn=bleater-ca`; agent regenerated with a different CN | The CA must have `Subject CN = bleater-ca` — `openssl req -x509 -subj "/CN=bleater-ca" ...`. The baseline config `/home/ubuntu/profile-mtls-baseline-config.txt` documents this. |
| Wrong SAN on regenerated leaf cert | grader.py uses `openssl s_client -verify_hostname`; agent's SAN doesn't include the headless FQDN | The leaf cert SAN must include **`bleater-profile-service-mtls.bleater.svc.cluster.local`** (not `bleater-profile-service...`). Check the `subjectAltName` in the openssl/cfssl config. |
| Trust bundle ConfigMap not repatched | Solution regenerated CA but forgot to update `bleater-trust-bundle` ConfigMap key `ca.crt` | `kubectl patch configmap bleater-trust-bundle -n bleater --type merge -p ...` with the new PEM-encoded CA |
| Secret not patched | Solution regenerated the leaf cert but forgot to write it to `bleater-profile-mtls` (keys `tls.crt`, `tls.key`) | `kubectl patch secret bleater-profile-mtls -n bleater --type merge ...` with the new material |
| `bleater-profile-cache-sync` still running | Strategy A deployment is still re-patching after solution.sh fixes the Secret | `kubectl scale deployment bleater-profile-cache-sync -n bleater --replicas=0` **before** cert regeneration |
| `node-telemetry-batcher` CronJob still active | kube-system CronJob fires every 60s mid-window | `kubectl delete cronjob node-telemetry-batcher -n kube-system` |
| Grader connects before pods are ready | Pods are still restarting when the grader probe fires | The `wait_consistent` retry loop handles this most of the time; if persistent, add `kubectl wait --for=condition=Ready pod -l app=... -n bleater --timeout=90s` after the delete |

### Branch D. Oracle `score: 0.5` — `trust_governance` failed (handshake passed)

Cert handshake recovered but at least one drift controller is still re-breaking trust during the grader window.

```bash
grep -iE "cronjob|cert-reaper|cache-sync|hpa|canary|drift|governance" /tmp/oracle_log.txt | head -30
```

Common causes, ordered by likelihood:

| Cause | Why | Fix in solution.sh |
|---|---|---|
| `bleater-profile-cert-rotator` CronJob not deleted/suspended | grader.py Layer 1 — must be absent OR `spec.suspend=true` | `kubectl delete cronjob bleater-profile-cert-rotator -n bleater` (and its `-n bleater` SA/Role/RoleBinding) |
| `cluster-trust-syncer` Deployment still running in kube-system | grader.py Layer 2 — must be `replicas=0` AND `readyReplicas=0` | `kubectl scale deployment cluster-trust-syncer -n kube-system --replicas=0` |
| `bleater-mesh-validator` Deployment still running in kube-system | grader.py Layer 2 — same | `kubectl scale deployment bleater-mesh-validator -n kube-system --replicas=0` |
| HPA pinner still active on `bleater-cert-reaper` | Solution scaled `bleater-cert-reaper` to 0 but `bleater-cert-reaper-pinner` HPA immediately restored `replicas=1` | `kubectl delete hpa bleater-cert-reaper-pinner -n bleater` **before** scaling. The `[11:04]` incident-notes line warns about this. |
| `bleater-cert-reaper` Deployment still running in bleater | grader.py Layer 2 — same as the kube-system entries (replicas=0, readyReplicas=0) | After deleting the HPA: `kubectl scale deployment bleater-cert-reaper -n bleater --replicas=0` |
| `bleater-profile-canary` mutation capability still intact | grader.py Layer 3 — needs sidecar OR `-runtime` RBAC removed (either is sufficient) | Easiest: `kubectl delete deployment bleater-profile-canary -n bleater` + delete the canary-runtime SA/Role/RoleBinding |
| Grader's `wait_consistent` window too short | A controller briefly dies and re-spawns inside the 40s window | Do **not** increase past `passes=5, gap=8s, timeout=200s` — at the ceiling, this is the wrong fix. Investigate which controller is respawning. |

### Branch E. Oracle `score: 0` with non-null feedback — both subscores failed

Both subscores failed cleanly (the grader ran end-to-end). Most likely setup.sh did its job correctly *but* solution.sh is broken in a way that misses both dimensions. Read the feedback for each subscore independently — it usually names the specific check that fired. Then follow Branch C for `mtls_handshake` and Branch D for `trust_governance` in parallel.

If Oracle returns `score: 0` repeatedly even after fixing what the feedback says, suspect a setup-side issue contaminating the solution run — re-run No-Op to isolate.

### Branch F. Grader crashed with a Python exception

```bash
grep -A 5 "Traceback\|Exception\|Error" /tmp/oracle_log.txt | head -40
```

| Exception | Likely cause | Fix in grader.py |
|---|---|---|
| `subprocess.CalledProcessError` | A `kubectl` invocation inside the grader returned non-zero | Add `check=False` to the `subprocess.run` and handle the error path explicitly |
| `json.JSONDecodeError` | `kubectl ... -o json` returned empty (resource not found) or non-JSON | Wrap `json.loads()` in try/except, treat parse failure as subscore=0 |
| `KeyError` on a dict access | grader assumed a key exists that's missing in some cluster states | Use `.get(key, default)` |
| `subprocess.TimeoutExpired` | A `kubectl wait` is hanging | Add a finite timeout to every `kubectl wait` inside the grader; treat timeout as subscore=0 |

These are grader bugs, not task-design issues — fix them in place. They are not in the hard-constraint list.

---

## Step 4 — Pre-fix safety checklist

Before applying any fix from Step 3, confirm:

1. **Dockerfile invariants still hold** — see `mtls-task-guardian` Step 0
2. **Syntax is clean** — `bash -n setup.sh && bash -n solution.sh && python3 -m py_compile grader.py`
3. **The proposed change is in `mtls-score-tuner`'s safe-lever table** if it touches scoring difficulty
4. **You can articulate the why** — "this fixes X because Y" — in one sentence

Then hand off:

| If your fix is | Hand off to |
|---|---|
| setup.sh fault-injection bug | `mtls-task-guardian` Step 3 (push) directly |
| solution.sh bug (Branch C/D) | `mtls-task-guardian` Step 3 directly — solution.sh is not a tuning surface |
| grader.py exception (Branch F) | `mtls-task-guardian` Step 3 directly |
| Anything affecting subscore difficulty | `mtls-score-tuner` for diagnosis first — do **not** edit grader without diagnosis |
| Anything you cannot map to a row above | `horizon-agentic-reviewer` to verify on a live container |

---

## Known non-issue patterns (do NOT fix these)

| Pattern | Why it isn't a bug |
|---|---|
| Quality check 18/20 (or 15/20 on a noisy day) | Four checks (`v1_feedback`, `behavior_in_task_documentation`, `behavior_in_tests`, sometimes `task_clarity`) are LLM hallucinations against the heredoc baseline config. Permanent and acceptable — see `mtls-task-guardian` Step 2 |
| No-Op returns `score: 0` with detailed feedback | This is the grader working correctly — feedback names the missing dimensions |
| `bleater-profile-cache-sync` not discovered by agents in the first batch | Strategy A is intentionally hidden — not in baseline config, not named in the prompt. Discovery cost is the design |
| `trust_governance` failing on rollouts because agents only audited `bleater` namespace | This is the kube-system discovery axis working as intended — do not "fix" it by hinting at kube-system in the prompt |

---

## Verification cheat sheet

| Hypothesis | Probe |
|---|---|
| "Setup.sh deployed every drift source the grader expects" | `grep -E "cert-rotator\|trust-syncer\|mesh-validator\|cert-reaper\|profile-canary\|cache-sync\|telemetry-batcher\|cert-reaper-pinner" /tmp/oracle_log.txt` — all eight names should appear (5 governance-graded + cache-sync + telemetry-batcher + HPA pinner) |
| "Solution.sh runs the scale-down before the cert regen" | Read solution.sh; the `kubectl scale ... --replicas=0` lines must appear above the cert regeneration block |
| "HPA delete happens before cert-reaper scale" | Read solution.sh; `kubectl delete hpa bleater-cert-reaper-pinner` must appear above `kubectl scale deployment bleater-cert-reaper` |
| "Pods bounced after Secret patch via `kubectl delete pod`" | Read solution.sh; `kubectl delete pod -l app=<profile-label>` AND `kubectl delete pod -l app=<bleat-label>` must appear after the Secret apply (NOT `rollout restart` — the actual solution.sh uses `delete pod`) |
| "Grader catches regenerated cert with wrong CN/SAN" | `grep -E "cn=bleater-ca\|verify_hostname" grader.py` — these MUST appear; they catch agents who regenerate the CA with the wrong identity. Their absence would be a regression. |
| "No baseline-preserve cert check has crept in" | `grep -iE "NotAfter\|NotBefore\|Issuer\|Serial" grader.py` — must return nothing (those are v6 dead-weight patterns) |
| "Wait_consistent params haven't drifted past ceiling" | `grep -E "wait_consistent" grader.py` — handshake exactly `passes=5, gap=15, timeout=240`; governance exactly `passes=5, gap=8, timeout=200`. Different values = drift. |
| "Setup.sh wipes events at the end" | `grep -n "kubectl delete events" setup.sh` — must appear before the hide-grader block (so agents can't reverse-engineer faults from event history) |