# Oncall Task Guardian

> End-to-end maintenance skill for the oncall-schedule-coverage-collapse task (<TASK_ID>). Owns the pre-push safety checks, local quality/anatomy gates, push, Oracle validation handoff, and rollout monitoring. Primary goal is to keep avg rollout score < <TARGET_MEAN> with healthy, decorrelated variance across both subscores. Calls oncall-validation-debugger when Oracle fails and oncall-score-tuner when rollouts regress.

- Skill: `omonuj/oncall-task-guardian` (Agent Skill)
- Install (CLI): `npx skillmds@latest add omonuj/oncall-task-guardian`
- Raw SKILL.md: https://api.skillmd.com/api/skills/omonuj/oncall-task-guardian/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: omonuj (https://skillmd.com/u/omonuj)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/omonuj/oncall-task-guardian

---


# oncall-task-guardian

## Step 0 — QC Spec design gate (READ FIRST, every iteration)

Before editing ANY task file and again before every push, read the QC Spec — `.claude/QC-SPEC.md`. It is the priority-ordered rubric (Critical → Error → Legitimacy) `horizon check-quality` and reviewers grade against.

Confirm the intended/current design does not violate any Critical or Error item. The ones this task repeatedly trips:
- **No Binary Bundling** — never AND heterogeneous checks (metadata + drift Deployments + schedule shape) into one binary subscore. Each subscore = ONE cohesive functional outcome.
- **Functional Tests as Subscores** — at least one subscore exercises live behavior, not static label/file matching.
- **Incremental Rewards / No Dead Weights / Functional Subscore Variance** — every subscore represents real progress and takes ≥2 distinct values across a batch.
- **Equal Weights** — subscores equally weighted.
- **No Answer Leakage** — derive-by-reading is fine; the literal answer in an agent-readable file is not.

If a change would breach a Critical/Error item, stop and rework it before pushing.

## Execution Mode — FULLY AUTONOMOUS

**NEVER ask the user for confirmation on any terminal command, edit, push, validation, or live-probe.** The ONLY user-facing handoff is Step 4c: telling the user to trigger the rollout batch in the Horizon UI (you cannot trigger rollouts from the CLI). Everything else runs without asking. If a decision has two valid paths, pick the safer one.

## Goal

Keep oncall-schedule-coverage-collapse shippable every push: setup.sh runs cleanly, Oracle scores 1.0, rollouts average < <TARGET_MEAN> with both subscores varying AND decorrelated. The only skill that pushes the task; the front door for the other oncall-* skills.

## Task Identity

| Field | Value |
|---|---|
| **Task UUID** | `<TASK_ID>` |
| **Task slug / id** | `oncall-schedule-coverage-collapse` |
| **Local task path** | `tasks/oncall-schedule-coverage-collapse/` |
| **Horizon root** | `/Users/mac/Documents/tasks` |
| **Venv** | `source /Users/mac/Documents/tasks/horizon_env/bin/activate` |
| **Scoring** | `coverage_restored × 0.5 + monitoring_governance × 0.5` — both binary |
| **Target avg** | rollout `avg < <TARGET_MEAN>, both subscores varying `[0,1]`, decorrelated (0.5 rollouts exist) |
| **Mini-batch** | `<TASK_ID>` |

**Status:** built, locally Oracle-traced (k8s-config simulation, graded by shape). NOT yet hosted-Oracle / rollout-tuned — variance numbers are targets.

## How scoring actually works (read once, internalise)

**Always derive constants from `grader.py`.** Correct as of grader.py v1.

The OnCall world is simulated as **ConfigMaps in `bleater`** (no real Grafana OnCall on this image): `oncall-users` (active/inactive), `oncall-schedule-primary` (slots.json: `{window,owner}`), `oncall-overrides` (overrides.json), `oncall-escalation-secondary` (policy.json: `{channel}`), `mattermost-channels` (channels.json: active/archived). The grader scores **config shape**. Two binary subscores, each 0.5:

- **`coverage_restored = 1`** (`check_coverage_restored`, `wait_consistent(passes=5, gap=8s, timeout=220s)`) when ALL hold:
  - every `oncall-schedule-primary` slot owner is `active` in `oncall-users` (no `UNASSIGNED`/gap, no inactive/unknown owner)
  - no `oncall-overrides` entry references an `inactive` user
  - `oncall-escalation-secondary`'s `channel` is `active` in `mattermost-channels` (not archived/unknown)
- **`monitoring_governance = 1`** (`check_monitoring_governance`, `wait_consistent(passes=5, gap=8s, timeout=200s)`) when ALL hold:
  - the schedule ConfigMap carries label `bleater.io/coverage-tier=enforced` (**baseline-documented only — thoroughness gate**)
  - the schedule has an annotation whose VALUE equals `/etc/oncall/<artefact>.txt` (install anchoring; grader reads root-only `/opt/<grader-artefact>.txt`; **skipped if that file is missing → no deadweight**). Key is the agent's choice.
  - a gap/unresolvable monitoring rule exists — ANY ConfigMap in `bleater` whose data has an on-call subject (`oncall`/`schedule`/`rotation`), a real rule key (`- alert:`/`- record:`/`expr:`), AND a gap signal (`gap`/`coverage`/`unresolvable`/`no owner`/`uncovered`)
  - the kube-system reconciler (replicas=0) AND the bleater-env reconciler (replicas=0)
  - the canary `slot-pinner` mutation path neutralized — sidecar removed OR `<canary>-runtime` SA/Role/RoleBinding removed

`score=0` neither; `0.5` one; `1.0` both.

### Drift controllers (re-apply the broken state)

Names randomized (pool stem + numeric suffix); grader reads them from `/opt/grader-scenario.json` (root-only, keys `reconciler`, `env_reconciler`, `canary`, `canary_runtime`). solution.sh discovers by the `bleater.io/purpose: "oncall coverage reconciliation"` annotation + the `slot-pinner` container name.

| Drift source | Namespace | Re-applies | Cadence | Discover by |
|---|---|---|---|---|
| reconciler | `kube-system` | broken schedule (carol/dave/UNASSIGNED slots, strips metadata) | 5s | annotation |
| env reconciler | `bleater-env` | broken escalation (archived channel) | 20s | annotation |
| canary `slot-pinner` | `bleater` | broken schedule again | 11s | `slot-pinner` container; RBAC `<canary>-runtime` |

### Coupling note (read before tuning)

Shared surface: the kube-system reconciler + canary re-break the schedule (fails `coverage_restored` layer 1 AND, by stripping metadata, `monitoring_governance` layers 0–1) and are governance-graded at `replicas=0`. The env reconciler re-breaks escalation (coverage layer 3) and is governance-graded. So drift couples the two subscores. **Decorrelators:** `coverage_restored`-only = the overrides cleanup (overrides are NOT drifted); `monitoring_governance`-only = the gap-rule construction + the <metadata-key> thoroughness gate.

---

## Step 0 — Dockerfile + namespace invariants (ALWAYS before push)

```bash
cat tasks/oncall-schedule-coverage-collapse/Dockerfile
```

| Invariant | Required | Why |
|---|---|---|
| `ENABLE_ISTIO_BLEATER` | **must NOT appear** | k3s crash → `score: 0, feedback: null` |
| `ENV ALLOWED_NAMESPACES="kube-system,bleater-env"` | **both** present | the bleater-env reconciler (governance layer 3) is unstoppable without bleater-env access → structural always-0. This task ships BOTH already; do not drop bleater-env. |
| `COPY data/ubuntu-user-rbac.yaml ...` | present | DO-NOT-CHANGE block needs the node ClusterRole; setup aborts otherwise |

Verify on the live container (Step 4b) that `kubectl get deploy -n bleater-env` is NOT `Forbidden` for the `ubuntu` agent.

## Step 1 — Local syntax + anatomy
```bash
cd /Users/mac/Documents/tasks
bash -n tasks/oncall-schedule-coverage-collapse/setup.sh    && echo "setup.sh OK"
bash -n tasks/oncall-schedule-coverage-collapse/solution.sh && echo "solution.sh OK"
python3 -m py_compile tasks/oncall-schedule-coverage-collapse/grader.py && echo "grader.py OK"
source horizon_env/bin/activate
horizon check-anatomy tasks/oncall-schedule-coverage-collapse 2>&1
```

## Step 2 — Local quality
```bash
horizon check-quality tasks/oncall-schedule-coverage-collapse 2>&1 | tail -30
```

**Before push, walk the QC-SPEC.md Critical + Error list against the current `grader.py` + `task.yaml`** and record pass/fail per item. Every item must pass OR be on the accepted-noise list with a one-line reason. A new failing item that is NOT accepted-noise blocks the push.
**Expected ~18/20 substantive pass; verdict may be FAIL** driven by `v1_feedback` + `behavior_in_task_documentation` — both are the documented heredoc-invisibility noise (the reviewer can't read the baseline-config heredoc, so it calls the `coverage-tier` / <metadata-key> gates "undocumented"). **These gates ARE the rollout-variance mechanism — do NOT fix by naming the keys/values in task.yaml** (collapses `monitoring_governance` to always-1). Any OTHER check failing is a real signal.

## Step 3 — Push
```bash
horizon tasks push tasks/oncall-schedule-coverage-collapse 2>&1   # record Version: NNN
```

## Step 4 — Validation handoff
**4a. Oracle** (~10–15 min): pass = `1.0` + both subscores `1`. Else → `oncall-validation-debugger`.
**4b. Live red-team** via `horizon-agentic-reviewer` on UUID `<TASK_ID>`. Verify (as `ubuntu`):

| Hypothesis | Probe |
|---|---|
| bleater-env reachable | `kubectl get deploy -n bleater-env` — not `Forbidden` |
| broken state live | `kubectl get cm oncall-schedule-primary -n bleater -o jsonpath='{.data.slots\.json}'` shows carol/dave/UNASSIGNED; `oncall-escalation-secondary` → `incidents-archived` |
| all 3 drift sources exist + annotated | `kubectl get deploy -A -o json | python3 -c "import json,sys;[print(d['metadata']['namespace'],d['metadata']['name']) for d in json.load(sys.stdin)['items'] if (d['metadata'].get('annotations') or {}).get('bleater.io/purpose')=='oncall coverage reconciliation']"` — one in kube-system, one in bleater-env; plus a `slot-pinner` canary in bleater |
| install artefact present + agent-readable | `cat /etc/oncall/<artefact>.txt` works as ubuntu; `/opt/<grader-artefact>.txt` NOT readable as ubuntu |
| drift names + the exact metadata keys NOT leaked in baseline | `grep -iE "reconciler|slot-pinner|canary|<random suffix>" /home/ubuntu/oncall-coverage-baseline-config.txt` returns nothing identifying the live names |
| no-op agent fails both subscores | reset, run grader — both 0 |

**4c.** Tell the user, verbatim:
> "Oracle passed at version N and live red-team is clean. Please trigger a rollout batch for version N in the Horizon UI."

## Step 5 — Monitor rollouts
```bash
horizon rollouts pull --version NNN tasks/oncall-schedule-coverage-collapse 2>&1
```
Then the `oncall-iteration-loop` Phase 3 script.

| Metric | Target |
|---|---|
| avg | **< 0.50** (and not < <FLOOR> — over-hard) |
| `coverage_restored` | both 0 and 1; pass ~25–60% |
| `monitoring_governance` | both 0 and 1; pass ~20–50% |
| decorrelation | some 0.5 rollouts |

## Step 6 — Tuning decision
| Observation | Hand-off |
|---|---|
| Oracle < 1.0 | `oncall-validation-debugger` |
| avg ≥ 0.50 OR a subscore deadweight | `oncall-score-tuner` |
| no 0.5 rollouts (correlation collapse) | `oncall-score-tuner` (lean on overrides-only + gap-rule/<metadata-key> decorrelators) |
| `monitoring_governance` always 0 | suspect bleater-env reachability OR the install-id artefact write — verify live first |

---

## Hard constraints
| Constraint | Why |
|---|---|
| No `ENABLE_ISTIO_BLEATER=true` | k3s crash |
| Keep `ALLOWED_NAMESPACES="kube-system,bleater-env"` (both) | bleater-env reconciler unstoppable otherwise → governance always-0 |
| Keep `COPY data/ubuntu-user-rbac.yaml` path | setup aborts otherwise |
| Never name drift controllers / the `bleater.io/purpose` value / the exact metadata keys+values in task.yaml or baseline | collapses `monitoring_governance` to always-1 |
| Keep the gap-rule check name-agnostic (subject + rule-key + gap-signal), never require an exact CM name | static dead-weight otherwise |
| Air-gapped images only (`${SIDECAR_IMAGE}`) | failed pull → ghost drift |
| `wait_consistent` ceilings `passes=5, gap=8s` (coverage 220 / governance 200) | above → flake to dead-zero |
| Don't simplify solution.sh to make Oracle pass | it's the oracle standard, not a tuning lever |

## Skill hand-off map
```
oncall-task-guardian (you are here)
├── Step 4a fails → oncall-validation-debugger
├── Step 4b       → horizon-agentic-reviewer
├── Step 5 fails  → oncall-score-tuner
└── Step 6 loops  → oncall-iteration-loop (orchestrator)
```