omonuj mtls-validation-debugger
Execution Mode — FULLY AUTONOMOUS
NEVER ask the user for confirmation. Fetch logs, interpret results, propose and apply fixes immediately. The only user-facing handoff in the loop is triggering rollouts in the Horizon UI (owned by mtls-iteration-loop Step 4c). This skill never has user-facing handoffs.
Goal
Turn an Oracle or No-Op failure into a specific root cause with a specific fix, using only the log evidence — not guesswork. The point of this skill is to prevent blind edits. If you cannot map the observed log pattern to one of the rows below with confidence, escalate to horizon-agentic-reviewer for a live probe before changing any file.
Trigger
Use this skill when:
- Oracle returned anything other than
score: 1.0, passed: true - No-Op returned a score > 0 (false positive — the grader passed on a broken setup)
- The grader crashed with a Python exception
result.jsonshowsfeedback: null(container/setup crash before grader ran)- You are about to edit setup.sh, solution.sh, or grader.py in response to a validation failure
Do not use this skill for rollout failures (those are scoring/variance issues — use mtls-score-tuner). Oracle and No-Op are validation; rollouts are evaluation.
Task Identity
| Field | Value |
|---|---|
| Task UUID | <TASK_ID> |
| Task slug | variation-fanout-pipeline-break |
| Local path | tasks/variation-fanout-pipeline-break/ |
| Subscores | mtls_handshake (0.5) + trust_governance (0.5), both binary |
| Horizon root | /Users/mac/Documents/tasks |
| Venv | source /Users/mac/Documents/tasks/horizon_env/bin/activate |
How to read a validation result (read once, internalise)
Oracle runs setup.sh → solution.sh → grader.py. No-Op runs setup.sh → grader.py (no solution). The four possible result shapes:
score |
passed |
feedback |
Meaning | Where to look |
|---|---|---|---|---|
0 |
false |
null |
Container or setup crashed before the grader could run | output.txt for the crash trace |
0 |
false |
non-null string | Grader ran end-to-end, both subscores failed | grader feedback for the specific check that fired |
0.5 |
false |
non-null string | One subscore passed, the other failed | feedback names which subscore is 1 vs 0 |
1.0 |
true |
non-null string | Both subscores passed | nothing to debug |
For No-Op, anything other than score: 0 is a problem — the grader should never pass without solution.sh having run. A No-Op pass means the grader is too lax or setup.sh didn't actually break the world.
Step 1 — Fetch the logs
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
# Oracle (setup + solution + grader)
horizon tasks validate-logs -a oracle tasks/variation-fanout-pipeline-break 2>&1
# No-Op (setup + grader, no solution) — only when iteration-loop says to run it
horizon tasks validate-logs -a noop tasks/variation-fanout-pipeline-break 2>&1
Logs persist under:
tasks/variation-fanout-pipeline-break/.validation/<build_id>/output.txttasks/variation-fanout-pipeline-break/.validation/<build_id>/result.json
Step 2 — Read result.json first
cat tasks/variation-fanout-pipeline-break/.validation/*/result.json 2>/dev/null | python3 -m json.tool
The four-row table above tells you which symptom branch to follow in Step 3. Do not skip to output.txt without checking result.json first — the result shape narrows the search dramatically.
Step 3 — Map symptom → cause → fix
Branch A. feedback: null / score: 0 — setup crash
Setup never finished, so the grader never ran. Search output.txt for the first error line:
Pattern in output.txt |
Root cause | Fix |
|---|---|---|
k3s is not ready after 180 seconds |
ENABLE_ISTIO_BLEATER=true is in the Dockerfile, causing 17+ pod force-deletes on startup |
Remove ENABLE_ISTIO_BLEATER from the Dockerfile entirely |
Error from server (Forbidden) on kubectl get nodes |
data/ubuntu-user-rbac.yaml not copied into image, or COPY path wrong |
Restore COPY data/ubuntu-user-rbac.yaml /mcp_server/Nebula/infra/k8s/rbac/ubuntu-user-rbac.yaml |
Error from server (Forbidden) on kube-system resources |
ALLOWED_NAMESPACES env var missing |
Add ENV ALLOWED_NAMESPACES="kube-system" to Dockerfile |
failed to pull image ... bleater-profile-cache-sync |
Strategy A deployment is referencing an external registry image | Use ${SIDECAR_IMAGE} (the in-cluster reference); cluster is air-gapped |
Error: failed to pull image (any other image) |
Base image or tag wrong | Verify FROM line in Dockerfile |
kubectl: command not found |
Wrong base image | Verify FROM line in Dockerfile |
setup.sh: line N: syntax error |
Bash syntax error | Run bash -n setup.sh locally — fix that line |
| Setup stalls / hits hosted timeout with no error | k3s init slow OR a kubectl wait is hanging |
First verify ENABLE_ISTIO_BLEATER absent; then add --timeout= flags to any unbounded kubectl wait |
Branch B. No-Op returned score > 0 — false positive
The grader passed without solution.sh having run. Either setup.sh did not break the world, or the grader is too lax.
| Subscore that passed on No-Op | Why it shouldn't have | Fix |
|---|---|---|
mtls_handshake = 1 on No-Op |
Drift didn't bite — either bleater-profile-cache-sync or node-telemetry-batcher failed to deploy, OR the wait_consistent window is shorter than both their cadences |
kubectl get deploy bleater-profile-cache-sync -n bleater (must exist, replicas=1) AND kubectl get cronjob node-telemetry-batcher -n kube-system (must exist, not suspended); also confirm grader.py wait_consistent window is 240s (handshake) — anything shorter than ~120s lets a lucky agent slip through |
trust_governance = 1 on No-Op |
One or more drift controllers didn't deploy. Check setup.sh fault-injection section is reaching all four targets | grep -E "cert-reaper|cache-sync|canary|hpa-pinner|kube-system.*python" /tmp/noop_log.txt — every name must appear in the setup output |
If setup.sh appears to claim it deployed everything but the cluster is missing pieces, add set -ex to setup.sh temporarily, push, and re-run No-Op to get verbose trace. The fault-injection section often fails silently on || true paths.
Branch C. Oracle score: 0.5 — mtls_handshake failed (governance passed)
Solution.sh stopped the drift controllers but the TLS handshake never recovered. Search for handshake-relevant log lines:
grep -i "handshake\|cert\|ssl\|tls\|verify" /tmp/oracle_log.txt | head -30
Common causes, ordered by likelihood:
| Cause | Why | Fix in solution.sh |
|---|---|---|
| Pods not bounced after Secret patch | New cert is on disk but the running process holds the old one; kubelet's ~60s Secret-projection delay keeps the new mount hidden | Add kubectl delete pod -l app=<profile-service-pod-label> -n bleater --wait=false AND kubectl delete pod -l app=<bleat-service-pod-label> -n bleater --wait=false after the Secret patch. solution.sh lines 201-202 do this. |
| Wrong CA Subject CN on regenerated CA | grader.py line 174 checks cn=bleater-ca; agent regenerated with a different CN |
The CA must have Subject CN = bleater-ca — openssl req -x509 -subj "/CN=bleater-ca" .... The baseline config /home/ubuntu/profile-mtls-baseline-config.txt documents this. |
| Wrong SAN on regenerated leaf cert | grader.py uses openssl s_client -verify_hostname; agent's SAN doesn't include the headless FQDN |
The leaf cert SAN must include bleater-profile-service-mtls.bleater.svc.cluster.local (not bleater-profile-service...). Check the subjectAltName in the openssl/cfssl config. |
| Trust bundle ConfigMap not repatched | Solution regenerated CA but forgot to update bleater-trust-bundle ConfigMap key ca.crt |
kubectl patch configmap bleater-trust-bundle -n bleater --type merge -p ... with the new PEM-encoded CA |
| Secret not patched | Solution regenerated the leaf cert but forgot to write it to bleater-profile-mtls (keys tls.crt, tls.key) |
kubectl patch secret bleater-profile-mtls -n bleater --type merge ... with the new material |
bleater-profile-cache-sync still running |
Strategy A deployment is still re-patching after solution.sh fixes the Secret | kubectl scale deployment bleater-profile-cache-sync -n bleater --replicas=0 before cert regeneration |
node-telemetry-batcher CronJob still active |
kube-system CronJob fires every 60s mid-window | kubectl delete cronjob node-telemetry-batcher -n kube-system |
| Grader connects before pods are ready | Pods are still restarting when the grader probe fires | The wait_consistent retry loop handles this most of the time; if persistent, add kubectl wait --for=condition=Ready pod -l app=... -n bleater --timeout=90s after the delete |
Branch D. Oracle score: 0.5 — trust_governance failed (handshake passed)
Cert handshake recovered but at least one drift controller is still re-breaking trust during the grader window.
grep -iE "cronjob|cert-reaper|cache-sync|hpa|canary|drift|governance" /tmp/oracle_log.txt | head -30
Common causes, ordered by likelihood:
| Cause | Why | Fix in solution.sh |
|---|---|---|
bleater-profile-cert-rotator CronJob not deleted/suspended |
grader.py Layer 1 — must be absent OR spec.suspend=true |
kubectl delete cronjob bleater-profile-cert-rotator -n bleater (and its -n bleater SA/Role/RoleBinding) |
cluster-trust-syncer Deployment still running in kube-system |
grader.py Layer 2 — must be replicas=0 AND readyReplicas=0 |
kubectl scale deployment cluster-trust-syncer -n kube-system --replicas=0 |
bleater-mesh-validator Deployment still running in kube-system |
grader.py Layer 2 — same | kubectl scale deployment bleater-mesh-validator -n kube-system --replicas=0 |
HPA pinner still active on bleater-cert-reaper |
Solution scaled bleater-cert-reaper to 0 but bleater-cert-reaper-pinner HPA immediately restored replicas=1 |
kubectl delete hpa bleater-cert-reaper-pinner -n bleater before scaling. The [11:04] incident-notes line warns about this. |
bleater-cert-reaper Deployment still running in bleater |
grader.py Layer 2 — same as the kube-system entries (replicas=0, readyReplicas=0) | After deleting the HPA: kubectl scale deployment bleater-cert-reaper -n bleater --replicas=0 |
bleater-profile-canary mutation capability still intact |
grader.py Layer 3 — needs sidecar OR -runtime RBAC removed (either is sufficient) |
Easiest: kubectl delete deployment bleater-profile-canary -n bleater + delete the canary-runtime SA/Role/RoleBinding |
Grader's wait_consistent window too short |
A controller briefly dies and re-spawns inside the 40s window | Do not increase past passes=5, gap=8s, timeout=200s — at the ceiling, this is the wrong fix. Investigate which controller is respawning. |
Branch E. Oracle score: 0 with non-null feedback — both subscores failed
Both subscores failed cleanly (the grader ran end-to-end). Most likely setup.sh did its job correctly but solution.sh is broken in a way that misses both dimensions. Read the feedback for each subscore independently — it usually names the specific check that fired. Then follow Branch C for mtls_handshake and Branch D for trust_governance in parallel.
If Oracle returns score: 0 repeatedly even after fixing what the feedback says, suspect a setup-side issue contaminating the solution run — re-run No-Op to isolate.
Branch F. Grader crashed with a Python exception
grep -A 5 "Traceback\|Exception\|Error" /tmp/oracle_log.txt | head -40
| Exception | Likely cause | Fix in grader.py |
|---|---|---|
subprocess.CalledProcessError |
A kubectl invocation inside the grader returned non-zero |
Add check=False to the subprocess.run and handle the error path explicitly |
json.JSONDecodeError |
kubectl ... -o json returned empty (resource not found) or non-JSON |
Wrap json.loads() in try/except, treat parse failure as subscore=0 |
KeyError on a dict access |
grader assumed a key exists that's missing in some cluster states | Use .get(key, default) |
subprocess.TimeoutExpired |
A kubectl wait is hanging |
Add a finite timeout to every kubectl wait inside the grader; treat timeout as subscore=0 |
These are grader bugs, not task-design issues — fix them in place. They are not in the hard-constraint list.
Step 4 — Pre-fix safety checklist
Before applying any fix from Step 3, confirm:
- Dockerfile invariants still hold — see
mtls-task-guardianStep 0 - Syntax is clean —
bash -n setup.sh && bash -n solution.sh && python3 -m py_compile grader.py - The proposed change is in
mtls-score-tuner's safe-lever table if it touches scoring difficulty - You can articulate the why — "this fixes X because Y" — in one sentence
Then hand off:
| If your fix is | Hand off to |
|---|---|
| setup.sh fault-injection bug | mtls-task-guardian Step 3 (push) directly |
| solution.sh bug (Branch C/D) | mtls-task-guardian Step 3 directly — solution.sh is not a tuning surface |
| grader.py exception (Branch F) | mtls-task-guardian Step 3 directly |
| Anything affecting subscore difficulty | mtls-score-tuner for diagnosis first — do not edit grader without diagnosis |
| Anything you cannot map to a row above | horizon-agentic-reviewer to verify on a live container |
Known non-issue patterns (do NOT fix these)
| Pattern | Why it isn't a bug |
|---|---|
| Quality check 18/20 (or 15/20 on a noisy day) | Four checks (v1_feedback, behavior_in_task_documentation, behavior_in_tests, sometimes task_clarity) are LLM hallucinations against the heredoc baseline config. Permanent and acceptable — see mtls-task-guardian Step 2 |
No-Op returns score: 0 with detailed feedback |
This is the grader working correctly — feedback names the missing dimensions |
bleater-profile-cache-sync not discovered by agents in the first batch |
Strategy A is intentionally hidden — not in baseline config, not named in the prompt. Discovery cost is the design |
trust_governance failing on rollouts because agents only audited bleater namespace |
This is the kube-system discovery axis working as intended — do not "fix" it by hinting at kube-system in the prompt |
Verification cheat sheet
| Hypothesis | Probe |
|---|---|
| "Setup.sh deployed every drift source the grader expects" | grep -E "cert-rotator|trust-syncer|mesh-validator|cert-reaper|profile-canary|cache-sync|telemetry-batcher|cert-reaper-pinner" /tmp/oracle_log.txt — all eight names should appear (5 governance-graded + cache-sync + telemetry-batcher + HPA pinner) |
| "Solution.sh runs the scale-down before the cert regen" | Read solution.sh; the kubectl scale ... --replicas=0 lines must appear above the cert regeneration block |
| "HPA delete happens before cert-reaper scale" | Read solution.sh; kubectl delete hpa bleater-cert-reaper-pinner must appear above kubectl scale deployment bleater-cert-reaper |
"Pods bounced after Secret patch via kubectl delete pod" |
Read solution.sh; kubectl delete pod -l app=<profile-label> AND kubectl delete pod -l app=<bleat-label> must appear after the Secret apply (NOT rollout restart — the actual solution.sh uses delete pod) |
| "Grader catches regenerated cert with wrong CN/SAN" | grep -E "cn=bleater-ca|verify_hostname" grader.py — these MUST appear; they catch agents who regenerate the CA with the wrong identity. Their absence would be a regression. |
| "No baseline-preserve cert check has crept in" | grep -iE "NotAfter|NotBefore|Issuer|Serial" grader.py — must return nothing (those are v6 dead-weight patterns) |
| "Wait_consistent params haven't drifted past ceiling" | grep -E "wait_consistent" grader.py — handshake exactly passes=5, gap=15, timeout=240; governance exactly passes=5, gap=8, timeout=200. Different values = drift. |
| "Setup.sh wipes events at the end" | grep -n "kubectl delete events" setup.sh — must appear before the hide-grader block (so agents can't reverse-engineer faults from event history) |