fanout-cve-validation-debugger
Execution Mode — FULLY AUTONOMOUS
NEVER ask for confirmation. Fetch logs, interpret, fix immediately. No user-facing handoffs. Can't map a pattern → escalate to horizon-agentic-reviewer.
Trigger
Oracle ≠ 1.0/passed; No-Op score > 0; grader exception; feedback: null (setup crash); or before editing setup/solution/grader. NOT for rollout-gradient issues — use fanout-cve-score-tuner.
Task Identity
| Field |
Value |
| UUID |
<TASK_ID> |
| Slug / path |
fanout-cve-rollout / tasks/fanout-cve-rollout/ |
| Subscores (0.2 each) |
cve_image_patched, configmap_reverted, pods_rolled_live_env, gitops_committed, argocd_synced |
| Venv |
source /Users/mac/Documents/tasks/horizon_env/bin/activate |
Grader feedback is name=0/1 (detail) for each subscore, joined by ; , plus a partial-fix HINT. Root-only file the grader trusts: /opt/grader-cve.json (patched_version, vulnerable_versions). Oracle = 1.0 only when all five are 1.
Step 1 — Fetch + read
cd /Users/mac/Documents/tasks && source horizon_env/bin/activate
horizon tasks validate-logs -a oracle tasks/fanout-cve-rollout 2>&1
cat tasks/fanout-cve-rollout/.validation/*/result.json 2>/dev/null | python3 -m json.tool
Step 2 — Map symptom → cause → fix
Branch A. feedback: null / score: 0 — setup crash
output.txt pattern |
Cause |
Fix |
k3s is not ready after 180 seconds |
ENABLE_ISTIO_BLEATER=true |
remove it |
Forbidden on kubectl get nodes |
rbac not copied |
restore COPY data/ubuntu-user-rbac.yaml |
failed to pull image for the fanout pod |
external registry referenced |
use ${SIDECAR_IMAGE} — the fanout pods MUST be runnable (the grader execs them) |
| fanout rollout never converges in setup |
image unrunnable / resource limits |
confirm ${SIDECAR_IMAGE} + sane limits |
setup.sh: line N: syntax error |
bash error |
bash -n setup.sh |
Branch B. No-Op score > 0 — false positive
| Subscore |
Why wrong on No-Op |
Fix |
configmap_reverted=1 |
setup didn't patch the CM to peer |
confirm the kubectl patch configmap ... peer runs AFTER the rollout |
pods_rolled_live_env=1 |
pods snapshotted peer not full, OR setup never created the drift |
setup must create the CM at full, roll the Deployment, THEN patch CM→peer (so live env=full while CM=peer) |
cve_image_patched=1 |
the Deployment shipped a clean version |
setup must label the template bleater.io/fanout-version=1.4.2 (vulnerable) |
gitops_committed=1 / argocd_synced=1 |
helm-values already patched / argocd already Synced |
setup must seed fanout-helm-values with the vulnerable version and argocd-bleater-app status: OutOfSync |
Branch C. Oracle partial (< 1.0) — one or more subscores failed
Read each name=0 (detail) and fix the matching solution.sh step:
| Failed subscore |
Cause |
Fix in solution.sh |
cve_image_patched |
template version label still vulnerable |
patch the Deployment template label bleater.io/fanout-version to the patched version (this also rolls the pods) |
configmap_reverted |
CM still peer |
kubectl patch configmap fanout-tls-config ... full |
pods_rolled_live_env + HINT fired |
pods not rolled, or rolled while CM was still peer |
revert the CM to full FIRST, then bump the version (rolls pods → they snapshot full); ensure kubectl rollout status converges before grading |
pods_rolled_live_env (no hint, version mismatch) |
a stale pod from the old ReplicaSet survived |
ensure the rollout fully converged (updated==ready==replicas) |
gitops_committed |
helm-values not updated |
apply fanout-helm-values with the patched fanoutVersion + amqpTlsValidationMode: full |
argocd_synced |
argocd status not Synced |
kubectl patch configmap argocd-bleater-app ... {"status":"Synced"} |
Branch D. Grader exception
The exec helper retries and tolerates failure (returns None → subscore 0 with a clear message). JSON/kubectl access is guarded. If a new crash appears, wrap it; treat failure as subscore=0.
Common ordering bug (the #1 oracle failure)
If the solution bumps the version (rolls pods) BEFORE reverting the ConfigMap, the new pods snapshot peer → pods_rolled_live_env fails on live env peer. Always revert the ConfigMap to full first, then roll.
Step 3 — Pre-fix safety
- Dockerfile invariants hold.
bash -n setup.sh && bash -n solution.sh && py_compile grader.py.- Subscore-design change → diagnose with
fanout-cve-score-tuner first.
- Articulate the why.
| Fix type |
Hand off |
| setup fault-injection / solution ordering / grader exception |
fanout-cve-task-guardian Step 3 (push) |
| subscore independence / partial-credit design |
fanout-cve-score-tuner |
| unmappable |
horizon-agentic-reviewer |
Known non-issues (do NOT fix)
| Pattern |
Why |
v1_feedback FAIL ("ConfigMap schemas missing") |
reviewer can't read the heredoc ConfigMap data. Accepted noise |
| Oracle partial during a fix iteration |
expected — the gradient is dense; read which subscore is 0 |
Verification cheat sheet
| Hypothesis |
Probe |
| "solution reverts CM before rolling" |
read solution.sh — the fanout-tls-config ... full patch precedes the version bump |
| "grader execs live pod env" |
`grep -E "exec |
| "grader trusts root-only CVE truth" |
`grep -E "/opt/grader-cve.json |
| "5 independent subscores, 0.2 each" |
`grep -E "weights |
omonuj