omonuj oncall-score-tuner
QC Spec compliance — assert before every tuning edit
Any difficulty/variance lever you pull must STILL satisfy the QC Spec (.claude/QC-SPEC.md). Before recommending a change, assert it does not breach:
- No Binary Bundling — never recover variance by AND-ing more heterogeneous checks into one subscore.
- Incremental Rewards / No Dead Weights — no subscore left always-0 or always-1; every lever keeps both subscores varying.
- Functional Subscore Variance — at least one functional subscore takes ≥2 distinct values across a batch.
- Equal Weights — never rebalance weights to hit a number.
- Functional Tests as Subscores / No Answer Leakage — don't convert a functional subscore into static matching, and don't leak the answer to manufacture passes.
A lever that lowers avg by breaching any of these is NOT a valid fix — pick a different lever.
Execution Mode — FULLY AUTONOMOUS
NEVER pause for confirmation. Pick the best path and execute it.
Goal
Keep avg < <TARGET_MEAN>, both subscores varying, decorrelated, no deadweight. Smallest reversible edits. Invoked on: avg ≥ 0.50; a subscore deadweight; correlation collapse (no 0.5 rollouts); oncall-task-guardian Step 6 hand-off. NOT for Oracle failures → oncall-validation-debugger.
Task Identity
| Field | Value |
|---|---|
| UUID | <TASK_ID> |
| Scoring | coverage_restored × 0.5 + monitoring_governance × 0.5 |
| Target | avg < <TARGET_MEAN>, both in {0,1}`, NOT perfectly correlated |
Status: not yet rollout-tuned. Watch the first batch for correlation collapse (the predicted failure mode).
How difficulty is constructed
OnCall state simulated as ConfigMaps in bleater. The agent repairs coverage + escalation + overrides, anchors metadata, adds a gap-detection rule, and stops three cross-namespace drift sources.
The shared surface
| Drift | Namespace | Re-applies | Cadence | Discover by |
|---|---|---|---|---|
| reconciler | kube-system | broken schedule (dead/UNASSIGNED slots, strips metadata) | 5s | bleater.io/purpose annotation |
| env reconciler | bleater-env | broken escalation (archived channel) | 20s | annotation |
canary slot-pinner |
bleater | broken schedule again | 11s | container name |
Both subscores depend on stopping this set → correlation risk. The kube-system reconciler/canary re-break the schedule (coverage layer 1 + governance metadata layers) and are governance-graded; the env reconciler re-breaks escalation (coverage layer 3) and is governance-graded.
coverage_restored axis (wait_consistent 5/8/220)
slots all active owners (no gap), no inactive override, escalation channel active. Independent decorrelator: the overrides cleanup — overrides are NOT drifted, so an agent who stops all drift still fails coverage if they leave carol/dave overrides.
monitoring_governance axis (wait_consistent 5/8/200)
coverage-tier label; annotation (== /etc/oncall/<artefact>.txt, grader reads /opt/<grader-artefact>.txt, skipped if missing); gap/unresolvable rule (real - alert:/expr: + on-call subject + gap signal); kube-system + bleater-env reconcilers replicas=0; canary neutralized. Independent decorrelators: the gap-rule construction (agents who hand-wave a CM with "gap" in a comment fail the - alert:/expr: requirement) and the thoroughness gate (read /etc/oncall/, ~40%-catch on the sibling pattern).
Structural deadweight check (do FIRST if governance always-0)
Confirm bleater-env reachability — ALLOWED_NAMESPACES must include bleater-env (it ships set; don't drop it) or the env reconciler is unstoppable → governance deadweight-0. Also confirm /opt/<grader-artefact>.txt was written (else the anchor gate silently skips, making governance easier, not harder).
Four rules
- Preserve variance. Stuck-at-one = broken. Always-0 → bleater-env reachability / install-id write. Always-1 → too discoverable.
- Break correlation. Prefer a lever moving ONE subscore: coverage → overrides; governance → gap-rule / . A drift-discoverability lever moves both — use only to shift avg.
- Functional > static. The gap-rule (real PromQL keyed by schedule) and (a value the agent must fetch from
/etc/oncall/) are durable. Never require an exact CM name; never leak the exact label/value to task.yaml. - Minimal safe edits.
Tuning order
Level 1 — setup.sh
Shared drift (move BOTH — shift avg only): reconciler 5s / env 20s / canary 11s cadences (faster = harder, floors 5/10/8); pool-name innocuousness (keep annotation-discoverable). Decorrelating (preferred): strengthen/weaken the overrides fault (coverage-only); the install-id artefact + baseline pointer wording (governance-only thoroughness); the gap-rule expectation phrasing (governance-only).
Level 2 — grader.py
wait_consistent (ceilings gap 8s; coverage timeout floor 180, governance 150). The gap-rule strictness (subject + rule-key + gap-signal) is the governance construction gate — keep functional, name-agnostic. Don't add a check that just matches a baseline-copyable value.
Level 3 — task.yaml (last resort)
Allowed: strengthen/weaken the "more than one namespace" hint; the "read baseline in full / inspect /etc/oncall" pointer. Forbidden: naming a drift controller, the annotation value, the exact metadata key/value, or itemizing the fix.
Diagnosis guide
| Symptom | Cause | Lever |
|---|---|---|
| no 0.5 rollouts (correlation) | drift is the only hard thing | lean on the overrides (coverage-only) + gap-rule/ (governance-only) decorrelators; don't pull a shared-drift lever |
coverage_restored always 1 |
drift easy + agents fix overrides | speed env/canary cadence; sharpen the overrides fault |
coverage_restored always 0 (Oracle 1.0) |
a drift source out-races the window | confirm cadences ≥ floor; verify bleater-env reachable |
monitoring_governance always 1 |
gap-rule + anchor too easy | keep the gap-rule strict; keep coverage-tier/anchor buried in baseline |
monitoring_governance always 0 (Oracle 1.0) |
bleater-env unreachable OR install-id mirror missing | structural — fix Dockerfile/setup, not grader |
| avg 0.60–0.70 | slightly easy | tighten a decorrelating gate, not a shared one |
Roles
- R1 Rollout Analyst — run the iteration-loop script, read ≥2 transcripts, classify (deadweight / healthy / correlated).
- R2 Setup Drift Specialist — which drift do failing agents miss? (bleater-env env-reconciler is the sneaky cross-namespace one.)
- R3 Coverage Specialist — overrides + escalation; the overrides cleanup is the coverage-only decorrelator.
- R4 Governance Specialist — gap-rule + + coverage-tier; rule out bleater-env reachability + install-id write first.
- R5 Coupling Reviewer (critical) — does the edit move ONLY shared drift? If yes → REJECT for decorrelation work; propose an overrides/gap-rule/anchor alternative.
- R6 QC Reviewer — expect 18 substantive pass +
v1_feedback/behavior_in_task_documentationnoise FAIL. Reject naming keys/values/drift in task.yaml; reject an exact-CM-name rule check. - R7 Final Recommender — 8-point block (root cause / why avg shifted / smallest fix / rollout+correlation impact / QC impact / risk / files / lines). Then →
oncall-iteration-loop→oncall-task-guardian.
Operating procedure
Confirm trigger → read grader/setup/task + rollouts + ≥2 transcripts → diagnose (R1) → pick lever (prefer decorrelating) → refine (R3/R4) → review (R5 + R6) → recommend (R7) → NEVER apply here; hand to the loop/guardian.
Hard constraints
No ENABLE_ISTIO_BLEATER; keep ALLOWED_NAMESPACES="kube-system,bleater-env"; keep the rbac COPY; never name drift/keys/values in task.yaml/baseline; keep the gap-rule functional & name-agnostic; air-gapped images; wait_consistent ceilings 5/8s (coverage 220 / governance 200); don't simplify solution.sh.
Preferred outcomes
Healthy: 0.0/0.5/1.0 present; avg –0.55; both vary AND decorrelated. Unhealthy: deadweight; avg > ; perfect correlation; structural impossibility (bleater-env unreachable, install-id mirror missing). Two cycles chasing the same Unhealthy pattern → halt and summarize.