# Ocas Custodian

> Monitors agent gateway logs, cron jobs, skill journals, and OCAS data directories for operational failures. Detects errors, applies safe non-destructive fixes autonomously during quiet hours, and escalates only what it cannot fix. Performs root cause analysis on recurring errors with fix-loop detection and confidence-tier auto promote/demote. Use when: cron jobs fail or show stale errors, gateway logs show repeated error patterns, skill journals have gaps, disk usage exceeds thresholds, MCP servers crash-loop, or after any gateway restart. Keywords: cron health, log analysis, system monitoring, error fingerprinting, auto-repair, fix-loop detection, operational conformance. NOT for OKR trend analysis, skill design evaluation, behavioral lesson extraction, briefing delivery, entity knowledge queries, or social graph queries.

- Skill: `indigokarasu/ocas-custodian` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add indigokarasu/ocas-custodian`
- Raw SKILL.md: https://api.skillmd.com/api/skills/indigokarasu/ocas-custodian/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Product & Planning
- License: MIT
- Author: indigokarasu (https://skillmd.com/u/indigokarasu)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/indigokarasu/ocas-custodian

---


# Custodian

Enforces the recovery contract defined in `spec-ocas-recovery.md` across all OCAS skills: every scheduled run must write an evidence record (including no-op runs with `not_activity_reason`), schedule gaps must trigger remedial passes, degraded mode must be explicit (not silent skip), and self-repair must include re-validation.

## Interactive Menu

When invoked interactively, present a two-level menu. See `references/interactive-menu.md` for the full menu structure.

## When to Use

- System health monitoring and alerting
- Skill library audits (conformance, freshness, coverage)
- Cron job health checks
- Log compaction and disk space monitoring
- After any major system change — verify integrity
## When NOT to Use

- Real-time monitoring (use heartbeat instead)
- Skill creation or modification (use Forge)
- Content generation or research
- User-facing task execution

## Known Code Fixes

### send_email.py Template Type Mismatch (`oc_vesper_template_missing`)
`send_email.py` only recognizes `job_search` as a valid template type.
Vesper delivery wrapper passes `vesper_evening` which gets rejected,
causing evening briefing email delivery failures (failover HTML writing still works).
Reference: `references/send-email-template-mismatch.md`.

### Wrong Path Prefix in Skill Scripts
Skill scripts that hardcode `$AGENT_ROOT/commons/...` instead of
`$AGENT_ROOT/profiles/<profile>/commons/...` cause `FileNotFoundError`
in cron context. The profile prefix must always be included.
Reference: `references/wrong-path-prefix-in-skill-scripts.md`.

## Critical Pitfalls

### Tool Quirks in Cron/Scheduled Context

See `references/cron-json-write-heredoc-variable-expansion-failure.md` for the single-quoted heredoc variable expansion failure pattern — confirmed 2026-06-24, produced corrupted journal files with literal `$(date)` in JSON.

- **Cron/scheduled tool-failure modes** — `read_file` dedup, pipe-to-interpreter, `write_file` failure, `execute_code` blocked, heredoc `$(date)`, and the `hermes cron` CLI path mismatch — are catalogued **with their fixes in the Error Handling table below**. Consult that table directly rather than re-deriving. One nuance the table omits: **use a UNIQUE `/tmp/` script filename** (timestamp/random suffix, e.g. `/tmp/cust_lights_20260708T1505.py`) — in concurrent cron contexts, sibling agents overwrite shared `/tmp/` paths; confirmed 2026-07-08 a sibling overwrote `/tmp/custodian_jobs.py`, resolved by renaming. The system emits a `_warning` on sibling modification — rename, don't ignore.
- **MCP server PIDs running but connection failing**: Processes can be alive yet fail TaskGroup connection handshake. Check process liveness before escalating.
- **state.db bloat pattern**: Expected size <1GB. If >1GB, check WAL size, then VACUUM or old message pruning. **Contextual threshold**: state.db commonly grows to 5-10GB in production. Flag as `oc_state_db_oversized` (Tier 2) when >1GB AND disk >80% — at lower disk usage, 5-10GB is acceptable operational cost. VACUUM feasibility: free_disk >= db_size is sufficient. If disk >80%, recommend message pruning instead of VACUUM. (2026-06-23)

- **Stale 503 pattern (2026-07-27):** Nous provider HTTP 503 on 7 jobs — stale. Issue `oc_provider_503_upstream_capacity` stays open; resolve only after `hermes cron run <id>` succeeds.
- **Skill update-wrapper failures (rebase-stuck / path-mismatch / merge-conflict batches)**: when multiple `:*:update` cron jobs fail simultaneously with git rebase/merge errors or legacy-helper `code 1` after repo sync — abort, reset to origin/main, verify HEAD==origin/main, rerun wrappers, then force-flip registry via `hermes cron run`. Full per-repo recipe + why each step is safe: `references/skill-update-rebase-conflict-batch-pattern.md`. Confirmed 2026-07-22 & 2026-07-27.ation-Runner Cron-Mode JSONL Workflow

When running `custodian.escalation-runner` as a cron job, all `issues.jsonl` and journal file mutations must use `terminal()` with heredoc — never `read_file` (corrupts JSONL) and never `execute_code` (blocked in cron). See the skill body for the reliable Python heredoc pattern.

## Responsibility Boundary

**Owns:** gateway log scanning and error fingerprinting, cron job registry health, skill journal completeness, OCAS data directory health, skill initialization, background task conformance, Tier 1 auto-repair, activity model and schedule optimization, escalation signaling, fix effectiveness tracking, confidence-based tier management, skill library hygiene (detection of stubs, nested .git, orphaned files).

**Does not own:** OKR trend analysis (Mentor), skill design evaluation (Mentor, Forge), behavioral lesson extraction (Praxis), briefing delivery (Vesper), social graph (Weave). Never modifies any file inside a skill package directory.

## Ontology types

Custodian operates on system health data (logs, config files, journal metadata, storage usage).

## Optional Skill Cooperation

- **Vesper** -- writes InsightProposals to proposals dir; Vesper reads from there.
- **Mentor** -- journals tagged `escalation_needed: true` are readable by Mentor heartbeat.


## Commands

- `custodian.init` -- create storage, register background tasks, build activity model
- `custodian.scan.light` -- tail gateway log, check cron registry, retry failed fixes, check uninitialized skills
- `custodian.scan.deep` -- full sweep (see references/deep-scan.md)
- `custodian.verify {fix_id}` -- verify fix outcome
- `custodian.repair.auto` -- apply all pending Tier 1 fixes
- `custodian.repair.plan` -- generate repair plan for Tier 2/3 issues
- `custodian.issues.list` -- list open issues
- `custodian.issues.resolve {issue_id}` -- mark resolved
- `custodian.status` -- emit SkillStatus JSON
- `custodian.schedule.show` -- display scan schedule
- `custodian.escalation-runner` -- process escalated Tier 3+ issues
- `custodian.update` -- self-update from GitHub
- `custodian.secrets.audit` -- scan configs/skills/scripts/plugins for inline plaintext secrets (API keys, tokens, client secrets, passwords) stored in the "wrong places" instead of the canonical store (`$HERMES_HOME/../<profile>/.env` loaded into `os.environ` at gateway start; `secrets.bitwarden.access_token_env`; MCP `headers` `${ENV}` indirection; `security.redact_secrets`). Read-only. De-dupes by secret value. See `references/secret-audit.md`.
- `custodian.secrets.remediate` -- plan (and with `--apply`, perform the safe subset of) the migration: move inline MCP `headers` to `${ENV}` indirection, never overwrite an existing `.env` key, back up every touched file. Credential-blob `.json` files and hardcoded `.py` literals are flagged as MANUAL steps (refactor to read `os.getenv`), not blind-edited. Re-run audit to confirm 0 inline hits. See `references/secret-audit.md`.

## Example

Typical light-scan invocation and its evidence record: running `custodian.scan.light` reads `jobs.json`, tails the gateway log for new errors since the last scan timestamp, fingerprints each, and writes an observation journal to `{agent_root}/commons/journals/ocas-custodian/YYYY-MM-DD/{run_id}.json`. On a clean scan (every error job transient) it returns `[SILENT]` *after* writing that journal — the journal is the proof the scan ran; `[SILENT]` only suppresses delivery noise.

## Confidence Model

See `references/confidence-model.md`. Key: `confidence_score = sample_confidence × success_rate`. Auto-promotes/demotes tiers based on fix history.

## Execution Loops

**Light Scan** (every heartbeat): Run the following checklist in order. Do not skip steps — each gates the next.

- [ ] 1. Read `jobs.json` (the profile cron registry at `$HERMES_HOME/../<profile>/cron/jobs.json`; NOT via `hermes cron list`, which reads the wrong path). **Parse robustly**: the registry is a top-level object whose job list lives under the key `"jobs"` (a list) — use `d.get("jobs", [])`. A few older/copied registry copies *do* wrap the list under `data.jobs`; only fall back to `d.get("data", {}).get("jobs", [])` if the top-level `"jobs"` key is **absent**. **CRITICAL FALSE-CLEAN GUARD**: if your parse yields `len(jobs) == 0`, re-inspect the raw file head (`head -c 600 jobs.json`) before concluding anything — a wrong key silently returns 0 jobs and risks a false `[SILENT]` on a misparse. Confirmed 2026-07-16: a parse using `data.get("jobs")` returned 0 jobs on a live registry that actually holds 148 (top-level `"jobs"`); the false-clean risk was caught only by re-inspecting the raw file.
- **Post-fix registry verification**: after resetting merge-conflict skill repos and re-running `update_skill.sh`, the `jobs.json` registry still shows stale `last_status=error` until the next scheduled execution. Force-flip with `hermes cron run <job_id>` and verify `last_status` flipped to `ok`.
- [ ] 2. Tail gateway log for new errors since last scan timestamp
    - **CRITICAL GATEWAY-TRACEBACK GAP (2026-07-22):** A jobs.json-only scan can report "clean" (0 actionable error jobs) while the gateway is throwing **recurring plugin-code tracebacks** that NEVER surface as a `jobs.json` `last_error`. These come from gateway-internal paths (e.g. `conversation_compression.py`, `chronicle/engine/store.py`), not from cron job scripts. Example signatures found 2026-07-22: `sqlite3.IntegrityError: CHECK constraint failed: actor IN (...)` in `chronicle/engine/store.py:append_event`, and `TypeError: ChronicleContextEngine.compress() got an unexpected keyword argument 'force'`. Both recurred 14–19× pre-restart, 0× post-restart. Detection procedure:
      - `grep -nE "Traceback|IntegrityError|TypeError|ERROR gateway" <gateway.log>` since the last gateway restart line (`Received SIGTERM` / `Starting Hermes Gateway`).
      - **WHICH `gateway.log` IS LIVE (2026-07-24):** there are TWO. `~/.hermes/logs/gateway.log` (root) is a **stale copy** (last entries June 2026 in this deployment) — grepping it returns pre-restart noise and MISSES live July tracebacks. The **live** log is `$HERMES_HOME/../<profile>/logs/gateway.log` (the one `verify_plugin_defect_postrestart.py` reads); plugin tracebacks also land in `~/.hermes/logs/errors.log`. Verify recency (`grep -oE "20[0-9]{2}-[0-9]{2}-[0-9]{2}" <file> | tail -3`) before trusting any `gateway.log`, and run the post-restart verifier against the LIVE path. Full recipe + confirmed case: `references/gateway-log-live-vs-stale-path-2026-07-24.md`.
      - Dedup signatures (exception class + in-plugin frame, e.g. `store.py:append_event` / `compress() got ... 'force'`), collect first/last timestamps.
      - Drop signatures already represented by an OPEN issue in `issues.jsonl` (grep fingerprint/issue_id) — don't double-track.
      - Drop single-shot / pre-restart-only signatures that have NOT recurred post-restart (noise).
      - For a NEW recurring signature with NO open issue: persist as `status: open`, `escalation_needed: true`, `user_gated: false`, `recommended_tier: 4` (plugin code defect — NOT credentials, NOT user-gated). Append race-safely (`cat >> issues.jsonl << 'PYEOF'`, one JSON object per line) — never a whole-file rewrite the top-of-hour `custodian:light` pass could clobber. Record `new_issues` in the journal.
    - See `references/gateway-log-traceback-gap-detection.md` for the reusable recipe + known-signature catalog. **VERIFIER FALSE-LIVE BUG (fixed 2026-07-24):** `verify_plugin_defect_postrestart.py` previously bucketed each signature hit against the *nearest preceding* restart, so logs with many restarts always reported `post_restart_total>0 → LIVE` (false positive) and would re-escalate resolved issues. It now uses the MOST RECENT restart only. Before acting on any LIVE verdict, independently confirm with `awk 'NR>=<last_restart_lineno>' <log> | grep -cE "..."` — if 0 but verifier says LIVE, the verdict is wrong. Detection recipe + regression note: `references/verify-plugin-defect-postrestart-false-live-bug.md`. **VERIFIER FALSE-DORMANT (inverse, confirmed 2026-07-25):** the verifier only counts signatures in its curated regex catalog; an uncatalogued or regex-mismatched signature reports `post=0 → DORMANT` even when actively recurring post-restart. Confirmed: `compress_force` recurred at `2026-07-24T16:51:12Z` (after the 15:25:08Z restart) but the verifier said `post=0`; independent `awk` proved 1 post-restart hit, so a `resolved` issue was wrongly left closed and had to be REOPENED. The raw-grep window in Step 2 is the AUTHORITATIVE live signal; the verifier is advisory only. Rule: if verifier says DORMANT but the raw-grep window contains the signature post-restart → treat as LIVE, reopen/escalate; do NOT trust DORMANT. Full recipe: `references/verify-plugin-defect-postrestart-false-dormant-2026-07-25.md`.
- [ ] 3. Fingerprint each new error (match against known patterns in `references/`)
- [ ] 4. Check failed fixes from previous scan — verify each fix held
- [ ] 5. Check uninitialized skills (missing storage, no background tasks registered)
    - This includes checking for missing data directories, journals directories, and config files. **Canonical paths are `{agent_root}/commons/data/{skill}/` and `{agent_root}/commons/journals/{skill}/`** (see `references/conformance.md` § Skill Initialization and `references/fix-missing-skill-data-directories-2026-06-30.md`). **⚠️ Pitfall — wrong-path guess:** do NOT check `skills/<skill>/commons/data/` or `skills/<skill>/commons/journals/` — those subdirs do not exist; every active skill would falsely report missing dirs and waste a check cycle. Confirmed 2026-07-14: a light scan guessed `skills/<skill>/commons/data` and flagged 32 false-missing dirs before correcting to `commons/data/<skill>` (all present).
    - See references/fix-missing-skill-data-directories-2026-06-30.md and references/fix-missing-skill-journals-directories-2026-06-30.md for fix procedures
    - **CRITICAL: cross-reference against active cron jobs before flagging.** A skill with missing data/journals dirs but NO cron jobs referencing it (via `skill:` or `skills:[]`) is uninitialized-but-unused — note it in the journal as info-only, do NOT escalate. Only flag for remediation if at least one active cron job depends on the skill. Confirmed 2026-07-01: 10xeng-autofix and skilllab had missing dirs but zero cron job references — not actionable.
    - **Re-derive the active-skill set from the LIVE jobs.json every scan** — union the `skills` (array) AND `skill` (scalar) fields across ALL job entries; treat any skill appearing ≥1 time as actively referenced. Confirmed 2026-07-07: a prior scan reported `browser-vision`, `generative-art-algorithms`, `generative-art-deployment`, `ocas-lucid` as "unused (no cron refs)" — but all four ARE referenced (by `art:studio` and the `lucid:*` jobs). The cross-ref had silently dropped them, so they were wrongly left un-remediated while `art:studio` ran with missing data/journals dirs. If a skill dir is missing AND the live set contains it, remediate — **but first check WHICH dir is missing** (see `references/dead-skill-ref-active-job-fix.md`): if the skill dir itself is absent because the skill was **archived/merged** (not merely uninitialized), do NOT recreate it — remove the dead `skill`/`skills[]` reference from the job (`skill: null`) instead. Recreating a fake skill directory is the wrong fix for an archived skill. Only `mkdir + config.json` when the skill dir exists but its `data/`/`journals/` subdirs are missing. **never trust a prior scan's 'unused' label**; always re-derive from the live file.
- [ ] 6. Check error jobs for script path blocks and `Path.home()` resolution issues
    - **DE-AGGREGATE identical wrapper messages.** When ≥2 error jobs share a bare `Script exited with code 1` (or any identical low-information) `last_error`, they are NOT one root cause. Enumerate each, read its `script` field, and determine the REAL per-job failure (run the script; inspect `sys.exit` paths; for subprocess wrappers, run the wrapped command). A no-op-by-design exit (no stderr) is `oc_cron_no_agent_exit_1_noop` (Tier 2 surface-only); a traceback is a real failure. Confirmed 2026-07-07: a prior scan collapsed three distinct jobs (`monitor:list` 403, `monitor:journals` no-op, `SearXNG` infra) into one bucket and omitted `monitor:journals`. See `references/no-agent-script-exit-1-deaggregation-pitfall.md`. Use `scripts/classify_error_jobs.py` to surface every ambiguous wrapper job with its `script` name.
    - **MASKED-EXIT-1 VARIANT (no output at all):** A no_agent wrapper may call the real worker as a subprocess and convert ANY non-zero to `sys.exit(1)` with NO stdout/stderr forwarded (see `monitor_list.py` → `tasks_monitor.py`). In that case `last_error` is the bare `Script exited with code 1` AND a live re-run of the wrapper shows empty output — the real traceback is HIDDEN. To de-aggregate, run the WRAPPED script directly (`python3 <skill>/scripts/<worker.py> --mode check`) and read ITS traceback. If the live signature differs from any resolved covering issue → it is a NEW failure with no open issue (Step 8b/8e gap); persist it. Confirmed 2026-07-14: `monitor:list` failed live with `KeyError: 'access_token'` — masked by the wrapper, referenced only by two already-resolved issues → gap closed by writing `oc_google_tasks_access_token_missing`. See `references/monitor-list-exit1-mask-gap.md` and `references/monitor-list-masked-keyerror-pitfall.md`.
- [ ] 7. Check for jobs not running (stale `last_run_at` vs expected schedule)
- **CRITICAL**: `next_run_at < now` alone is NOT sufficient. Must verify: `last_run_at` older than 2× schedule interval AND `last_status != ok`. High-frequency jobs (≤10 min) show scheduler state lag where `next_run_at` hasn't advanced but job ran successfully. See `references/scheduler-state-lag-vs-execution-failure.md`.
- **Never-run job (`last_status=None` AND `last_run_at=None`):** NOT a stuck-scheduler failure if `next_run_at` is still in the future — the job has simply never been due. Only flag as "not running" when `next_run_at` is ALSO past. Convert `next_run_at` to UTC before comparing (it carries an offset, e.g. `-07:00`; see `references/jobs-json-timestamp-offset-misread-pitfall.md`). Confirmed 2026-07-22: `skill-sync-all` showed `last_status=None` / `last_run_at=None` with `next_run_at=2026-07-23T04:00-07:00` (next day) — correctly left unflagged.
- [ ] 8. For each fingerprint with `recurrence_count >= 2`, check `rca.jsonl` — if no RCA record exists, flag for deep scan RCA step; if Pattern B, skip fix and note in journal
- **8b. Journal-to-issues gap check**: for any previous journal entry with `escalation_needed: true`, verify a matching entry exists in `issues.jsonl` for the same fingerprint. If NOT found, write it — the prior scan flagged but failed to persist. (Confirmed pattern: 10:05 scan wrote `escalation_needed: true` but did NOT write the issue to issues.jsonl; 12:04 scan had to write it manually.) **STALE-PREMISE GUARD (2026-07-14):** before writing a gap issue, VERIFY THE LIVE PREMISE is still true — re-scan `jobs.json`, disk usage, and provider state. A journal flagged `escalation_needed: true` can carry a premise that resolved AFTER the journal was written; persisting it creates a FALSE escalation (pollutes `issues.jsonl` and burns an execution-loop cycle). Concrete checks: for `oc_state_db_oversized` re-derive disk% live (`shutil.disk_usage('/root')`) — threshold is db>1GB AND disk>80%; if disk is now `<80%` (even at 5-10GB db) it is acceptable operational cost, do NOT persist. **PATH TRAP (confirmed 2026-07-25):** the canonical state.db is `$HERMES_HOME/../<profile>/state.db` (~14.7GB in production) — the root `~/.hermes/state.db` is a 38-byte placeholder and will make you wrongly conclude the oversized premise is FALSE. Measure the PROFILE path (`ls -la $HERMES_HOME/../<profile>/state.db` plus `find ~/.hermes -name '*.db' -size +1G` to locate all >1GB DBs) before applying the stale-premise guard. A scan that checks only the root path almost failed to persist a genuine 84.9%-disk / 14.7GB-DB oversized condition. For any `*_access_token_missing` / auth fingerprint, if the implicated job is `status=ok` with cleared `last_error`, it recovered — do NOT persist. For every fingerprint, require ≥1 live job still matching the signature before writing. Confirmed 2026-07-14: `oc_state_db_oversized` (journal claimed disk 82%) was NOT persisted because live disk was 70.2%; `oc_google_tasks_access_token_missing` was NOT persisted because `monitor:list` was already `status=ok`. See `references/journal-escalation-stale-premise-guard-2026-07-14.md`.
- **8b-variant — 'tracked' in narrative but `escalation_needed: false`**: A scan may reference root-cause fingerprints as 'tracked' (in `previous_scan_delta.stable_root_cause` or prose) while setting `escalation_needed: false` on the journal. If the referenced fingerprint is absent from `issues.jsonl`, the escalation silently dropped — every later scan re-reports it as 'tracked' without ever persisting it. Fix: after classifying non-auto-fixable root causes, collect their intended fingerprints and verify each exists in `issues.jsonl` (use `scripts/parse_issues_jsonl.py`); if missing, WRITE it (`status: open`, `escalation_needed: true`). One issue per root-cause fingerprint; list affected job names in `affected_components`. Confirmed 2026-07-07: 71-job 402-credits and 2-job OAuth revocations were reported 'tracked' in prior deltas but absent from `issues.jsonl`; light scan wrote them. See `references/escalation-persistence-gap.md`.
- **8b/8e parser gotcha (confirmed 2026-07-14):** When reading `issues.jsonl` for gap checks, a hand-written brace-walk parser inside `python3 << 'PYEOF'` that does quote/backslash tracking returns **0 objects** (its escape handling breaks on `\"`). Use `json.JSONDecoder().raw_decode` instead — robust to escaped quotes and concat-per-line objects. In concurrent cron contexts (sibling `custodian:light` rewrites the file at top of hour) re-verify the file `mtime`/`size` across 3 quick reads before trusting a parse; prefer `grep -ac` for a single targeted check (e.g. does this `issue_id` already exist) to avoid a full race-prone rewrite. Race-safe recipe in `references/monitor-list-exit1-mask-gap.md`.
- [ ] 8f. **Recurrence-of-fingerprint-CLASS across distinct jobs (gap Step 8b/8b-variant does NOT cover):** The journal→issues gap checks only fire when a journal *flagged* `escalation_needed: true` but no open issue exists. They do NOT catch a live error job whose **fingerprint class** was seen before but whose prior issue was correctly *resolved per-job* (not per-fingerprint-family). Consequence: a same-class error silently recurs on a NEW job with no open issue and no flag, and every scan re-classifies it as "pre-classified" / "known pattern" without persisting anything. **Procedure:** for each live error job (Steps 6–8), map its error to a reusable fingerprint *class* (e.g. `content_policy_blocked`, `token_expired`, `402 credits`, `interpreter shutdown`, `Script exited with code 1`). Grep the full `issues.jsonl` (resolved + open) for that class token. If the class was previously seen but ALL matching issues are now `status: resolved`/`duplicate` AND the *current* job is **not** listed in any of those issues' `affected_job_ids` → this is a NEW occurrence of a recurring class on a distinct job → **persist a fresh issue** (`status: open`, `escalation_needed: true`, `user_gated` per class nature, `affected_job_ids` = [current job id]) even though no journal flagged it. Do NOT re-open the old resolved issue (it was correct for its own job); write a new dated issue. STALE-PREMISE GUARD still applies: require ≥1 live enabled job currently matching the signature before writing. Confirmed 2026-07-23: `ocas-autobio-observe` errored live with `content_policy_blocked`; prior same-class issues `oc_bones_content_policy_blocked_20260720` and `oc_sands_evening_brief_content_policy_20260722` were both `resolved` (Mentor note "re-escalate if recurrence detected") and listed only their own jobs — no open issue covered `ocas-autobio-observe`, so a new issue `oc_autobio_content_policy_blocked_20260723T0505Z` was written. See `references/recurrence-fingerprint-class-distinct-job-2026-07-23.md`.
- [ ] 8c. **Verify-before-accepting-self-resolved**: when a prior scan classified an error as "self-resolved" (e.g., `ModuleNotFoundError` that supposedly fixed itself), verify by running the actual import in the cron execution python — NOT any assumed venv path. Cron jobs run `python3` from PATH. To find the actual python: `which python3` in a terminal, then `python3 -c "import <module>"`. The profile venv path (`$HERMES_HOME/../<profile>/venv/bin/python3`) may NOT exist — the system hermes venv (`<hermes-venv>/bin/python3`) is typically the active one. Confirmed 2026-07-01: `dispatch:triage-morning` was classified "self-resolved" but the verification was done by checking the actual import (import google.oauth2.credentials → OK). Do NOT accept "self-resolved" from a prior journal entry without re-verifying when: (a) the module path in the error differs from what you assumed, (b) the error was from a no_agent script (different python resolution), or (c) the prior scan has known counting discrepancies. See `references/self-resolved-module-verification-pattern.md`.
- [ ] 8d-variant — 503 scope-expansion gap: If `oc_provider_503_upstream_capacity` exists but covers fewer live 503 jobs than erroring, record the gap in the journal under `scope_gaps` and flag for Mentor to expand `affected_job_ids`. Do NOT write a duplicate issue — one fingerprint = one issue. See `references/503-scope-expansion-2026-07-28.md`.
- [ ] 8e. **Verify resolved CODE-DEFECT fixes actually cover all references** (catches what `reopen_false_resolutions.py` misses): that script only matches provider/auth/credit outage signatures (`token_expired`, `402 credits`, `owl-alpha 404`) — it does NOT catch a `resolved` issue whose fingerprint is a code defect (`oc_*_bug`, `oc_*_missing`, `oc_*_path_*`) whose fix was incomplete. For each such `resolved` issue: (a) does any enabled job's `last_error` STILL contain the original error signature? If yes, first compare the job's `last_run_at` against the fix timestamp — if the job ran BEFORE the fix landed, the error is STALE (re-run the script to confirm), not a live regression; (b) does the source file the fix touched STILL contain the broken reference the error named? A "resolved" entry whose fix only added a comment saying 'we now use X' while the erroring line still references old Y is a FALSE resolution — reopen (`status: open`, `escalation_needed: true`, `tier: 4`, clear `resolved_at`, set `reopened_at` + `reopen_note`). Grep recipe: `grep -rn "<broken_token>" <file>`. Re-run the script directly (`<hermes-venv>/bin/python <script>` or `bash <wrapper>`) and inspect exit code + stderr to separate stale from live. **CORRECTED 2026-07-13 (supersedes the original misread assertion):** The original reopen directive for `oc_chronicle_facts_fts_missing_20260713` was a MISREAD and must NOT be followed. It claimed `enrich_embeddings.py:121` still executed `DELETE FROM facts_fts`, but line 121 is a COMMENT; the executable statements (lines 130/149) use the live `belief_fts`/`observed_fts` tables, which exist in `chronicle.db` and rebuild cleanly (verified by dry-executing both DELETEs against the live DB — no `OperationalError`). The job's stored error was STALE: last ran 2026-07-13 10:02 UTC, BEFORE the fix landed (file mtime 11:36 UTC). The resolution was CORRECT — leave it resolved. **Grep-pitfall (root cause of the misread):** `grep -rn "facts_fts" enrich_embeddings.py` matches only the historical comment lines (120, 124), producing a false "still broken" hit. A token appearing in comments is NOT proof the broken code path runs — read the actual executable lines around the cited line number, or execute the real code path, before reopening. Always compare `last_run_at` vs the fix file's mtime: an error from a run BEFORE the fix is stale, not a live regression. **FALSE-CLOSE via drained-backlog test (inverse, confirmed 2026-07-13):** When a prior scan *resolved* a timeout/volume code-defect issue on the strength of a live re-run that completed fast, verify that re-run executed against the REAL production backlog — NOT a queue just drained by a prior run. A timeout issue can finish in ~163s when there is nothing left to embed, then HANG past the 600s cron hard limit once daily volume rebuilds. In this session `oc_script_timeout_chronicle_embed_20260713` was closed at 21:36Z citing a 163s pass, but its `embed_state.json` showed `last_run=21:35:56` (a prior run had just drained the queue), and a fresh re-run at 23:0xZ was still actively embedding at 85s (row 4160/8000 of the facts pass; `facts` table holds 35,486 rows) — confirming the timeout is recurring. Before accepting a timeout/throughput 'resolved': (1) check the progress/state file's `last_run` vs the claimed fix time — if within seconds, the test ran on a cleared queue; (2) inspect real data volume (`SELECT COUNT(*) FROM facts`); (3) re-run the actual script against that full volume with a hard cap (background `terminal(background=true, notify_on_complete=true)` + `process(wait/poll)` — foreground cap is 60s) and confirm it completes under the cron limit. A clean run immediately after another successful run proves nothing about steady-state load. See `references/resolved-codefix-regression-verify.md` and `references/resolved-timeout-verify-drained-backlog.md`.
- [ ] 9. **Verify-before-acting**: for any error job, check current `config.yaml` and provider state to confirm the error is still active before attempting fix
- [ ] 10. Write observation journal (even if no issues found — set `not_activity_reason`)
- **10b. LLM-necessity guard**: Run `scripts/classify_llm_necessity_integration.py` to detect new cron jobs whose prompts don't need LLM reasoning (script-wrappers, self-updates, needless skill-load). This uses the acknowledgment file (`data/llm_necessity_ack.json`) to avoid re-reporting already-triaged candidates. Writes/updates one `oc_cron_llm_unnecessary` issue in `issues.jsonl` for new unacknowledged candidates. **REPORT-ONLY — never auto-converts a job to `no_agent`.** Run via: `python3 <hermes-home>/profiles/indigo/skills/ocas-custodian/scripts/classify_llm_necessity_integration.py`.

**Cron silence protocol:** When running as a scheduled cron job, if the scan finds no actionable issues, respond with exactly `[SILENT]`. Only produce a report when there is genuinely new information.

**Journal-before-silent requirement:** The recovery contract (see `spec-ocas-recovery.md`) requires every scheduled run to write an evidence record. Even a no-op scan with no actionable issues MUST write an observation journal (with `not_activity_reason` set) before returning `[SILENT]`. The correct sequence is: (1) write the journal → (2) return `[SILENT]`. Do NOT skip the journal on silent runs. The journal proves the scan ran; `[SILENT]` prevents unnecessary delivery noise.

**Deep Scan** (optimized 6h cron): Full 13-step sweep. See `references/deep-scan.md` and `references/deep-scan-2026-06-28-clean-verdict.md` for the clean verdict pattern (all-transient → journal + silent).

**Deep Scan early-exit shortcut:** When all error jobs are transient (cf=0/None, last_run before recent restart, no new fingerprints, no consecutive_failures >= 1), skip Steps 3b/4/5/9. Go directly to classification + Tier 1 fix pass. Trigger: all jobs have `consecutive_failures` in (0, None) AND all `last_error` match transient patterns (futures shutdown, exit 1 no-op, gateway collision, 429, script-not-found race, no_agent path mismatch, gateway restart import window, provider error transient). Do NOT skip journal or conformance checks. See `references/deep-scan.md`.

**Deep Scan clean verdict (2026-06-23):** When ALL error jobs classify as transient/non-faulty (known patterns, cf=None/0, no active issue), the scan is clean. No Tier 1 fixes needed. Write observation journal with `not_activity_reason` and return `[SILENT]`. This is the expected steady-state — a clean scan means the system is healthy, not that the scan missed something. Do not force-fix non-issues. Confirmed 2026-06-23: 16 error jobs, 100% transient (futures shutdown, 429 rate limit, no_agent exit 1 noop), 0 fixes applied, all clean.

**Delta journal for repeated clean verdicts:** When consecutive scans find the SAME errors with no new/resolved issues, journal a `previous_scan_delta` block: elapsed min, new_issues=0, new_errors=0, stable root cause. New transient errors alongside stable escalated issues: MAY still use delta but include `new_errors: N` + `new_error_detail`. Do NOT use after gateway restart, fix, or state change. See `references/light-scan-2026-06-29-0904.md`.

- **Fix-loop already escalated — don't re-escalate:** If fix-loop RCA Pattern B exists AND prior esc-run already escalated → note fingerprint in journal, do NOT re-fix, do NOT duplicate escalation, return `[SILENT]` with `not_activity_reason: "clean_verdict_all_errors_already_escalated"`. See `references/deep-scan-fix-loop-prehandled-silent-verdict.md`.

**Config empty section: "Tier 1 auto-fix" vs Pattern B contradiction:** `oc_config_empty_section` is Tier 1 auto-fix BUT has Pattern B RCA. Resolution: DO remove null keys (fixes TUI warnings), note fix-loop in journal, write escalation for architectural root cause (gateway regenerates null keys on restart). See `references/config-empty-section-fixloop-status.md`.

**Escalation Runner Checklist:**
- [ ] 1. Check latest esc-run journal first (5-sec check vs 60-sec full scan)
- [ ] 1b. **Already-classified fast path**: If the prior esc-run journal is < 2h old AND classified all open issues as `open_user_gated` AND no new `last_error` messages appear in `jobs.json` that weren't in the prior journal AND no new entries appear in any `issues.jsonl` → write journal referencing the prior classification and return `[S Skip Steps 2-6. See `references/escalation-runner-already-classified-fast-path.md`. Do NOT use after gateway restart, after applying a fix, or when a significant state change occurred.
- [ ] 2. Discover ALL `issues.jsonl` paths: `find <hermes-home> -name "issues.jsonl"`
- [ ] 3. Deduplicate by `issue_id`/`id` — keep best status per entry
- [ ] 4. For each open issue, verify against raw file (`terminal(command="cat ...")`) — not `custodian_issues` tool (stale cache)
- [ ] 5. Classify into four buckets: Actionable / User-gated / Legacy-inactive / Already-resolved
- [ ] 6. If any Actionable issues exist → execute fixes
- [ ] 7. If no Actionable issues → write journal with `not_activity_reason`, return `[SILENT]`
- [ ] 8. Clear `escalation_needed` flag on any resolved entries

**Escalation runner journal write — use Python always**: When running `escalation-runner` in cron context, write journals via `python3 -c "..."` with `json.dump()` and `from datetime import datetime, timezone; datetime.now(timezone.utc)`. Do NOT use `cat > file << 'EOF'` heredoc for JSON containing timestamps/run_ids — single-quoted heredoc prevents `$(date)` expansion, producing corrupted files. **Import the CLASS, not the module:** `import datetime; datetime.now()` raises `AttributeError: module 'datetime' has no attribute 'now'`. Always use `from datetime import datetime, timezone` so `datetime.now(timezone.utc)` resolves to the class method. See `references/escalation-runner-already-classified-fast-path.md` § Journal write pattern reminder.

### The fos→os typo in custodian scripts (confirmed 2026-07-27, NOW FIXED): `fos.path.expanduser` (should be `os.path.expanduser`) appeared in **8 files** across the custodian `scripts/` directory — a copy-paste error from an earlier module. Affected files: `classify_error_jobs.py` (lines 20, 90), `confirm_provider_recovery.py` (134), `verify_provider_recovery.py` (24, 40), `verify_escalation_state.py` (161), `find_missed_user_gated_jobs.py` (187), `bucket_error_jobs.py` (138), `classify_llm_necessity.py` (27), `classify_llm_necessity_integration.py` (24, 25, 273). Impact: `classify_error_jobs.py` crashes block Step 6 de-aggregation of exit-1 wrapper jobs; `classify_llm_necessity_integration.py` crashes block Step 10b LLM-necessity integration. Fix: `s/fos\.path\.expanduser/os.path.expanduser/g` across all files. If one occurrence is fixed, check ALL files in the skill's `scripts/` directory — it is a copy-paste error that recurs. See `references/fos-nameerror-pattern.md` for full details and impact analysis. **Resolved 2026-07-27: all 8 files patched, verified 0 remaining occurrences.**

**Escalation runner clean verdict — actionable vs user-gated vs legacy (2026-06-25):** When the escalation runner finds no actionable issues, classify all open entries into four buckets: (A) Actionable — execute fix; (B) User-gated — note count but do not auto-fix (skill library hygiene, stub removal); (C) Legacy/inactive — ignore YAML debris in profiles with no `cron/jobs.json`; (D) Already resolved — verify config/job state and close. If Bucket A is empty, write journal with `not_activity_reason` and return `[SILENT]`. See `references/escalation-runner-clean-verdict-pattern.md` for the decision tree and journal template. Inactive profile detection: check for `cron/jobs.json` absence (>90 days dormant). Confirmed 2026-06-25: braun profile has 3 null keys but no cron jobs — legacy debris, not an action item.

**Escalation runner: user-gated provider failures are not permanent kill switches (2026-07-09):** Do NOT pause recurring cron jobs just because their last error is provider auth/credits/429/endpoint outage. First verify the provider/model live with a minimal `hermes chat -q` probe. If the provider works now, resume affected jobs and let them retry; stale `last_error` is not live failure. Only pause when retry is genuinely futile until user action (e.g. revoked Google OAuth for a domain-specific tool, missing script, blocked execute_code redesign), and then write `pause_reason`, `jobs_paused`, journal evidence, and a re-enable-on-recovery check. Pausing is mitigation, never resolution. Detailed procedure: `references/stale-provider-error-pause-loops.md`.

**Escalation Execution Loop (external cron trigger) — EXECUTE, don't just classify (2026-07-07):** When an external loop invokes Custodian + Mentor to *execute* fixes on escalated issues (not merely classify them), the default Escalation Runner Checklist above is classification-oriented — this fills the execute-and-reconcile gap:

See `references/escalation-loop-pitfalls.md` for chronic traps in execution-loop runs: the journal-gap probe reports FALSE gaps for already-`resolved` issues (it compares only against OPEN issues), stale `last_error` vs live failure (inspect the wrapper/script + fix timestamp before reopening), cooperating with an in-flight sanctioned sibling repair instead of racing it, and reading the traceback location (not just the final exception) when a `database is locked` error appears after the root-cause step already passed.
1. **Verify live state BOTH directions** against `jobs.json` — (a) issue claims resolved/paused but job still `enabled`+erroring (inverse gotcha), AND (b) issue flags `escalation_needed: true` but the job already recovered (`last_status: ok`, `last_error` cleared). Resolve or re-pause accordingly; never trust the issue flag alone. AND (c) **sweep for missed enrollments**: any `enabled`+erroring job whose `last_error` matches a known user-gated fingerprint (Nous 401 `portal.nousresearch.com`, OpenRouter 402 `credits`, owl-alpha 404, Google Tasks 403, `invalid_grant`) but is NOT in any issue's `jobs_paused`. These failed in the inter-scan window *after* the last esc pass and were never enrolled. For provider/model fingerprints, verify the provider live first; if it now works, leave/resume the jobs and clear stale issue state 

…(truncated)