omonuj fanout-cve-score-tuner
QC Spec compliance — assert before every tuning edit
Any gradient/difficulty lever you pull must STILL satisfy the QC Spec (.claude/QC-SPEC.md). Before recommending a change, assert it does not breach:
- No Binary Bundling — keep the 5 subscores independent; never AND them or short-circuit the grade on one failure.
- Incremental Rewards / No Dead Weights — every 0.2 subscore must stay independently reachable and missed by some — none always-0 or always-1.
- Functional Tests as Subscores — keep
pods_rolled_live_enva livekubectl execcheck; never relax it to a spec read. - Equal Weights — all five subscores stay at 0.2.
- No Answer Leakage — CVE truth stays in root-only
/opt/grader-cve.json, never an agent-editable artefact.
A lever that improves the gradient by breaching any of these is NOT valid — pick a different lever.
Execution Mode — FULLY AUTONOMOUS
NEVER pause for confirmation. Pick the best path and execute.
What this skill optimizes (different from the variance tasks)
This is a reward-shaped task. There is NO avg < variance target, no drift, no decorrelation problem. The objective is a dense, correct, monotone-ish partial-credit gradient: the agent should earn 0.2 per genuinely-correct sub-step and only reach 1.0 with the complete two-phase remediation. Your job is to keep that gradient honest.
Invoked when: a subscore is deadweight (always-0 unreachable / always-1 free); partial credit is structurally impossible; a subscore is reward-hackable; the gradient mis-guides (e.g. you can score high while the system is still degraded); fanout-cve-task-guardian Step 5 hands off. NOT for Oracle failures → fanout-cve-validation-debugger.
Task Identity
| Field | Value |
|---|---|
| UUID | <TASK_ID> |
| Subscores (0.2 each) | cve_image_patched, configmap_reverted, pods_rolled_live_env, gitops_committed, argocd_synced |
The five subscores and what each must keep true
| Subscore | Reward-shaping invariant |
|---|---|
cve_image_patched |
reachable only by changing the version off the vulnerable set; judged off root-only /opt/grader-cve.json (NOT the agent-editable harbor-scan-results) — keep it that way or it becomes reward-hackable |
configmap_reverted |
a pure config check (AMQP_TLS_VALIDATION_MODE==full); independent of the others |
pods_rolled_live_env |
the keystone — must use live kubectl exec, not the pod spec, or "no restart" passes for free. Must require rollout convergence + clean version label + live env full. This is intentionally the hardest/most-missed subscore |
gitops_committed |
the Gitea source declares a clean version + full; independent of the cluster CM |
argocd_synced |
argocd-bleater-app status==Synced; independent |
Health checks (run these on the rollout distribution)
| Healthy | Unhealthy → action |
|---|---|
every score in {0,0.2,…,1.0} appears |
a value never appears → a subscore may be coupled/unreachable |
| each subscore passed by some agents, missed by some | deadweight subscore → fix (see below) |
pods_rolled_live_env is the most-missed |
if it's the LEAST-missed, the live-env check may be too weak (verify it execs) |
| a fair share reach 1.0 | nobody reaches 1.0 → an ordering/structural impossibility (check the revert-then-roll dependency) |
Levers (smallest first)
Deadweight cve_image_patched (always-0)
The version marker is unreachable. Confirm the agent can change bleater.io/fanout-version (it's a template label, not in the selector) and that harbor-scan-results + baseline name the patched version. Don't require an exact string the agent can't discover.
Deadweight pods_rolled_live_env (always-0, Oracle 1.0)
Likely the exec is failing in the rollout window. Confirm ${SIDECAR_IMAGE} has sh; add exec retries (already 2). Do NOT relax to a spec-only check.
pods_rolled_live_env always-1 (free)
The live-env check isn't biting — verify it execs the pod (not reads the spec configMapKeyRef, which always shows the reference). If the stale snapshot coincidentally equals the target, ensure the version-label convergence still forces a roll.
Partial credit impossible (only 0.0 and 1.0 appear)
The subscores have become coupled (AND-bundled). Keep them five independent functions returning independent 0/1 — never short-circuit the whole grade on one failure.
Reward-hack: agent edits harbor-scan-results to fake "clean"
Ensure cve_image_patched reads /opt/grader-cve.json (root-only), never the ConfigMap.
Mis-guidance: high score while still degraded
The danger case the author called out: scoring well without actually fixing the live system. Guard it — pods_rolled_live_env (live env + converged + clean version) is what ties the score to the real running state. Never let configmap_reverted + gitops_committed + argocd_synced (3×0.2=0.6, all structural) be achievable while the pods are still vulnerable/stale. They can be, individually — that's fine for shaping — but 1.0 must require the live pods correct.
Difficulty (medium — do NOT inflate)
The issue specifies medium. Do not add drift, cross-namespace discovery, hidden /etc artefacts, or admission webhooks — those belong to the variance tasks and would break the reward-shaping design and the difficulty target.
Roles
- R1 Distribution Analyst — pull rollouts, tabulate the score histogram + per-subscore pass rate; classify each subscore (deadweight / healthy / hackable).
- R2 Keystone Specialist — focus on
pods_rolled_live_env: is it execing live env, requiring convergence + clean version? Is it the right difficulty (most-missed but reachable)? - R3 Independence Reviewer — confirm the 5 subscores are independent functions; no AND-bundling; partial credit flows.
- R4 Anti-Hack Reviewer — confirm version judged off
/opt/grader-cve.json; no subscore passable by editing a read-only-intended artefact. - R5 Final Recommender — 8-point block (which subscore/gradient problem / why / smallest fix / expected distribution impact / QC impact / risk / files / lines). Then →
fanout-cve-iteration-loop→fanout-cve-task-guardian.
Operating procedure
Confirm trigger → read grader/setup/solution + rollouts → R1 distribution → identify the gradient problem → pick the smallest lever (keep medium, keep independence, keep anti-hack) → R3/R4 review → R5 recommend → NEVER apply here; hand to the loop/guardian.
Hard constraints
No ENABLE_ISTIO_BLEATER; keep the rbac COPY; judge version off /opt/grader-cve.json; keep the live-env exec in the keystone subscore; keep 5 independent subscores (no bundling); no drift/variance mechanics; medium difficulty; air-gapped runnable images (pods must exec); don't simplify solution.sh.
Preferred outcome
The full {0,0.2,0.4,0.6,0.8,1.0} gradient appears across a batch; pods_rolled_live_env is the most-instructive (most-missed but reached by careful agents); 1.0 requires the live pods to be patched + full; no subscore is deadweight or reward-hackable.