Ocas Custodian

Custodian

Enforces the recovery contract defined in spec-ocas-recovery.md across all OCAS skills: every scheduled run must write an evidence record (including no-op runs with not_activity_reason), schedule gaps must trigger remedial passes, degraded mode must be explicit (not silent skip), and self-repair must include re-validation.

Interactive Menu

When invoked interactively, present a two-level menu. See references/interactive-menu.md for the full menu structure.

When to Use

• System health monitoring and alerting
• Skill library audits (conformance, freshness, coverage)
• Cron job health checks
• Log compaction and disk space monitoring
• After any major system change — verify integrity

When NOT to Use

• Real-time monitoring (use heartbeat instead)
• Skill creation or modification (use Forge)
• Content generation or research
• User-facing task execution

Known Code Fixes

send_email.py Template Type Mismatch (oc_vesper_template_missing)

send_email.py only recognizes job_search as a valid template type. Vesper delivery wrapper passes vesper_evening which gets rejected, causing evening briefing email delivery failures (failover HTML writing still works). Reference: references/send-email-template-mismatch.md.

Wrong Path Prefix in Skill Scripts

Skill scripts that hardcode $AGENT_ROOT/commons/... instead of $AGENT_ROOT/profiles/<profile>/commons/... cause FileNotFoundError in cron context. The profile prefix must always be included. Reference: references/wrong-path-prefix-in-skill-scripts.md.

Critical Pitfalls

Tool Quirks in Cron/Scheduled Context

See references/cron-json-write-heredoc-variable-expansion-failure.md for the single-quoted heredoc variable expansion failure pattern — confirmed 2026-06-24, produced corrupted journal files with literal $(date) in JSON.

• Cron/scheduled tool-failure modes — read_file dedup, pipe-to-interpreter, write_file failure, execute_code blocked, heredoc $(date), and the hermes cron CLI path mismatch — are catalogued with their fixes in the Error Handling table below. Consult that table directly rather than re-deriving. One nuance the table omits: use a UNIQUE /tmp/ script filename (timestamp/random suffix, e.g. /tmp/cust_lights_20260708T1505.py) — in concurrent cron contexts, sibling agents overwrite shared /tmp/ paths; confirmed 2026-07-08 a sibling overwrote /tmp/custodian_jobs.py, resolved by renaming. The system emits a _warning on sibling modification — rename, don't ignore.

• MCP server PIDs running but connection failing: Processes can be alive yet fail TaskGroup connection handshake. Check process liveness before escalating.

• state.db bloat pattern: Expected size <1GB. If >1GB, check WAL size, then VACUUM or old message pruning. Contextual threshold: state.db commonly grows to 5-10GB in production. Flag as oc_state_db_oversized (Tier 2) when >1GB AND disk >80% — at lower disk usage, 5-10GB is acceptable operational cost. VACUUM feasibility: free_disk >= db_size is sufficient. If disk >80%, recommend message pruning instead of VACUUM. (2026-06-23)

• Stale 503 pattern (2026-07-27): Nous provider HTTP 503 on 7 jobs — stale. Issue oc_provider_503_upstream_capacity stays open; resolve only after hermes cron run <id> succeeds.

• Skill update-wrapper failures (rebase-stuck / path-mismatch / merge-conflict batches): when multiple :*:update cron jobs fail simultaneously with git rebase/merge errors or legacy-helper code 1 after repo sync — abort, reset to origin/main, verify HEAD==origin/main, rerun wrappers, then force-flip registry via hermes cron run. Full per-repo recipe + why each step is safe: references/skill-update-rebase-conflict-batch-pattern.md. Confirmed 2026-07-22 & 2026-07-27.ation-Runner Cron-Mode JSONL Workflow

When running custodian.escalation-runner as a cron job, all issues.jsonl and journal file mutations must use terminal() with heredoc — never read_file (corrupts JSONL) and never execute_code (blocked in cron). See the skill body for the reliable Python heredoc pattern.

Responsibility Boundary

Owns: gateway log scanning and error fingerprinting, cron job registry health, skill journal completeness, OCAS data directory health, skill initialization, background task conformance, Tier 1 auto-repair, activity model and schedule optimization, escalation signaling, fix effectiveness tracking, confidence-based tier management, skill library hygiene (detection of stubs, nested .git, orphaned files).

Does not own: OKR trend analysis (Mentor), skill design evaluation (Mentor, Forge), behavioral lesson extraction (Praxis), briefing delivery (Vesper), social graph (Weave). Never modifies any file inside a skill package directory.

Ontology types

Custodian operates on system health data (logs, config files, journal metadata, storage usage).

Optional Skill Cooperation

• Vesper -- writes InsightProposals to proposals dir; Vesper reads from there.
• Mentor -- journals tagged escalation_needed: true are readable by Mentor heartbeat.

Commands

• custodian.init -- create storage, register background tasks, build activity model
• custodian.scan.light -- tail gateway log, check cron registry, retry failed fixes, check uninitialized skills
• custodian.scan.deep -- full sweep (see references/deep-scan.md)
• custodian.verify {fix_id} -- verify fix outcome
• custodian.repair.auto -- apply all pending Tier 1 fixes
• custodian.repair.plan -- generate repair plan for Tier 2/3 issues
• custodian.issues.list -- list open issues
• custodian.issues.resolve {issue_id} -- mark resolved
• custodian.status -- emit SkillStatus JSON
• custodian.schedule.show -- display scan schedule
• custodian.escalation-runner -- process escalated Tier 3+ issues
• custodian.update -- self-update from GitHub
• custodian.secrets.audit -- scan configs/skills/scripts/plugins for inline plaintext secrets (API keys, tokens, client secrets, passwords) stored in the "wrong places" instead of the canonical store ($HERMES_HOME/../<profile>/.env loaded into os.environ at gateway start; secrets.bitwarden.access_token_env; MCP headers ${ENV} indirection; security.redact_secrets). Read-only. De-dupes by secret value. See references/secret-audit.md.
• custodian.secrets.remediate -- plan (and with --apply, perform the safe subset of) the migration: move inline MCP headers to ${ENV} indirection, never overwrite an existing .env key, back up every touched file. Credential-blob .json files and hardcoded .py literals are flagged as MANUAL steps (refactor to read os.getenv), not blind-edited. Re-run audit to confirm 0 inline hits. See references/secret-audit.md.

Example

Typical light-scan invocation and its evidence record: running custodian.scan.light reads jobs.json, tails the gateway log for new errors since the last scan timestamp, fingerprints each, and writes an observation journal to {agent_root}/commons/journals/ocas-custodian/YYYY-MM-DD/{run_id}.json. On a clean scan (every error job transient) it returns [SILENT] after writing that journal — the journal is the proof the scan ran; [SILENT] only suppresses delivery noise.

Confidence Model

See references/confidence-model.md. Key: confidence_score = sample_confidence × success_rate. Auto-promotes/demotes tiers based on fix history.

Execution Loops

Light Scan (every heartbeat): Run the following checklist in order. Do not skip steps — each gates the next.

• 1. Read jobs.json (the profile cron registry at $HERMES_HOME/../<profile>/cron/jobs.json; NOT via hermes cron list, which reads the wrong path). Parse robustly: the registry is a top-level object whose job list lives under the key "jobs" (a list) — use d.get("jobs", []). A few older/copied registry copies do wrap the list under data.jobs; only fall back to d.get("data", {}).get("jobs", []) if the top-level "jobs" key is absent. CRITICAL FALSE-CLEAN GUARD: if your parse yields len(jobs) == 0, re-inspect the raw file head (head -c 600 jobs.json) before concluding anything — a wrong key silently returns 0 jobs and risks a false [SILENT] on a misparse. Confirmed 2026-07-16: a parse using data.get("jobs") returned 0 jobs on a live registry that actually holds 148 (top-level "jobs"); the false-clean risk was caught only by re-inspecting the raw file.
• Post-fix registry verification: after resetting merge-conflict skill repos and re-running update_skill.sh, the jobs.json registry still shows stale last_status=error until the next scheduled execution. Force-flip with hermes cron run <job_id> and verify last_status flipped to ok.
• 2. Tail gateway log for new errors since last scan timestamp
  • CRITICAL GATEWAY-TRACEBACK GAP (2026-07-22): A jobs.json-only scan can report "clean" (0 actionable error jobs) while the gateway is throwing recurring plugin-code tracebacks that NEVER surface as a jobs.json last_error. These come from gateway-internal paths (e.g. conversation_compression.py, chronicle/engine/store.py), not from cron job scripts. Example signatures found 2026-07-22: sqlite3.IntegrityError: CHECK constraint failed: actor IN (...) in chronicle/engine/store.py:append_event, and TypeError: ChronicleContextEngine.compress() got an unexpected keyword argument 'force'. Both recurred 14–19× pre-restart, 0× post-restart. Detection procedure:
    • grep -nE "Traceback|IntegrityError|TypeError|ERROR gateway" <gateway.log> since the last gateway restart line (Received SIGTERM / Starting Hermes Gateway).
    • WHICH gateway.log IS LIVE (2026-07-24): there are TWO. ~/.hermes/logs/gateway.log (root) is a stale copy (last entries June 2026 in this deployment) — grepping it returns pre-restart noise and MISSES live July tracebacks. The live log is $HERMES_HOME/../<profile>/logs/gateway.log (the one verify_plugin_defect_postrestart.py reads); plugin tracebacks also land in ~/.hermes/logs/errors.log. Verify recency (grep -oE "20[0-9]{2}-[0-9]{2}-[0-9]{2}" <file> | tail -3) before trusting any gateway.log, and run the post-restart verifier against the LIVE path. Full recipe + confirmed case: references/gateway-log-live-vs-stale-path-2026-07-24.md.
    • Dedup signatures (exception class + in-plugin frame, e.g. store.py:append_event / compress() got ... 'force'), collect first/last timestamps.
    • Drop signatures already represented by an OPEN issue in issues.jsonl (grep fingerprint/issue_id) — don't double-track.
    • Drop single-shot / pre-restart-only signatures that have NOT recurred post-restart (noise).
    • For a NEW recurring signature with NO open issue: persist as status: open, escalation_needed: true, user_gated: false, recommended_tier: 4 (plugin code defect — NOT credentials, NOT user-gated). Append race-safely (cat >> issues.jsonl << 'PYEOF', one JSON object per line) — never a whole-file rewrite the top-of-hour custodian:light pass could clobber. Record new_issues in the journal.
  • See references/gateway-log-traceback-gap-detection.md for the reusable recipe + known-signature catalog. VERIFIER FALSE-LIVE BUG (fixed 2026-07-24): verify_plugin_defect_postrestart.py previously bucketed each signature hit against the nearest preceding restart, so logs with many restarts always reported post_restart_total>0 → LIVE (false positive) and would re-escalate resolved issues. It now uses the MOST RECENT restart only. Before acting on any LIVE verdict, independently confirm with awk 'NR>=<last_restart_lineno>' <log> | grep -cE "..." — if 0 but verifier says LIVE, the verdict is wrong. Detection recipe + regression note: references/verify-plugin-defect-postrestart-false-live-bug.md. VERIFIER FALSE-DORMANT (inverse, confirmed 2026-07-25): the verifier only counts signatures in its curated regex catalog; an uncatalogued or regex-mismatched signature reports post=0 → DORMANT even when actively recurring post-restart. Confirmed: compress_force recurred at 2026-07-24T16:51:12Z (after the 15:25:08Z restart) but the verifier said post=0; independent awk proved 1 post-restart hit, so a resolved issue was wrongly left closed and had to be REOPENED. The raw-grep window in Step 2 is the AUTHORITATIVE live signal; the verifier is advisory only. Rule: if verifier says DORMANT but the raw-grep window contains the signature post-restart → treat as LIVE, reopen/escalate; do NOT trust DORMANT. Full recipe: references/verify-plugin-defect-postrestart-false-dormant-2026-07-25.md.
• 3. Fingerprint each new error (match against known patterns in references/)
• 4. Check failed fixes from previous scan — verify each fix held
• 5. Check uninitialized skills (missing storage, no background tasks registered)
  • This includes checking for missing data directories, journals directories, and config files. Canonical paths are {agent_root}/commons/data/{skill}/ and {agent_root}/commons/journals/{skill}/ (see references/conformance.md § Skill Initialization and references/fix-missing-skill-data-directories-2026-06-30.md). ⚠️ Pitfall — wrong-path guess: do NOT check skills/<skill>/commons/data/ or skills/<skill>/commons/journals/ — those subdirs do not exist; every active skill would falsely report missing dirs and waste a check cycle. Confirmed 2026-07-14: a light scan guessed skills/<skill>/commons/data and flagged 32 false-missing dirs before correcting to commons/data/<skill> (all present).
  • See references/fix-missing-skill-data-directories-2026-06-30.md and references/fix-missing-skill-journals-directories-2026-06-30.md for fix procedures
  • CRITICAL: cross-reference against active cron jobs before flagging. A skill with missing data/journals dirs but NO cron jobs referencing it (via skill: or skills:[]) is uninitialized-but-unused — note it in the journal as info-only, do NOT escalate. Only flag for remediation if at least one active cron job depends on the skill. Confirmed 2026-07-01: 10xeng-autofix and skilllab had missing dirs but zero cron job references — not actionable.
  • Re-derive the active-skill set from the LIVE jobs.json every scan — union the skills (array) AND skill (scalar) fields across ALL job entries; treat any skill appearing ≥1 time as actively referenced. Confirmed 2026-07-07: a prior scan reported browser-vision, generative-art-algorithms, generative-art-deployment, ocas-lucid as "unused (no cron refs)" — but all four ARE referenced (by art:studio and the lucid:* jobs). The cross-ref had silently dropped them, so they were wrongly left un-remediated while art:studio ran with missing data/journals dirs. If a skill dir is missing AND the live set contains it, remediate — but first check WHICH dir is missing (see references/dead-skill-ref-active-job-fix.md): if the skill dir itself is absent because the skill was archived/merged (not merely uninitialized), do NOT recreate it — remove the dead skill/skills[] reference from the job (skill: null) instead. Recreating a fake skill directory is the wrong fix for an archived skill. Only mkdir + config.json when the skill dir exists but its data//journals/ subdirs are missing. never trust a prior scan's 'unused' label; always re-derive from the live file.
• 6. Check error jobs for script path blocks and Path.home() resolution issues
  • DE-AGGREGATE identical wrapper messages. When ≥2 error jobs share a bare Script exited with code 1 (or any identical low-information) last_error, they are NOT one root cause. Enumerate each, read its script field, and determine the REAL per-job failure (run the script; inspect sys.exit paths; for subprocess wrappers, run the wrapped command). A no-op-by-design exit (no stderr) is oc_cron_no_agent_exit_1_noop (Tier 2 surface-only); a traceback is a real failure. Confirmed 2026-07-07: a prior scan collapsed three distinct jobs (monitor:list 403, monitor:journals no-op, SearXNG infra) into one bucket and omitted monitor:journals. See references/no-agent-script-exit-1-deaggregation-pitfall.md. Use scripts/classify_error_jobs.py to surface every ambiguous wrapper job with its script name.
  • MASKED-EXIT-1 VARIANT (no output at all): A no_agent wrapper may call the real worker as a subprocess and convert ANY non-zero to sys.exit(1) with NO stdout/stderr forwarded (see monitor_list.py → tasks_monitor.py). In that case last_error is the bare Script exited with code 1 AND a live re-run of the wrapper shows empty output — the real traceback is HIDDEN. To de-aggregate, run the WRAPPED script directly (python3 <skill>/scripts/<worker.py> --mode check) and read ITS traceback. If the live signature differs from any resolved covering issue → it is a NEW failure with no open issue (Step 8b/8e gap); persist it. Confirmed 2026-07-14: monitor:list failed live with KeyError: 'access_token' — masked by the wrapper, referenced only by two already-resolved issues → gap closed by writing oc_google_tasks_access_token_missing. See references/monitor-list-exit1-mask-gap.md and references/monitor-list-masked-keyerror-pitfall.md.
• 7. Check for jobs not running (stale last_run_at vs expected schedule)
• CRITICAL: next_run_at < now alone is NOT sufficient. Must verify: last_run_at older than 2× schedule interval AND last_status != ok. High-frequency jobs (≤10 min) show scheduler state lag where next_run_at hasn't advanced but job ran successfully. See references/scheduler-state-lag-vs-execution-failure.md.
• Never-run job (last_status=None AND last_run_at=None): NOT a stuck-scheduler failure if next_run_at is still in the future — the job has simply never been due. Only flag as "not running" when next_run_at is ALSO past. Convert next_run_at to UTC before comparing (it carries an offset, e.g. -07:00; see references/jobs-json-timestamp-offset-misread-pitfall.md). Confirmed 2026-07-22: skill-sync-all showed last_status=None / last_run_at=None with next_run_at=2026-07-23T04:00-07:00 (next day) — correctly left unflagged.
• 8. For each fingerprint with recurrence_count >= 2, check rca.jsonl — if no RCA record exists, flag for deep scan RCA step; if Pattern B, skip fix and note in journal
• 8b. Journal-to-issues gap check: for any previous journal entry with escalation_needed: true, verify a matching entry exists in issues.jsonl for the same fingerprint. If NOT found, write it — the prior scan flagged but failed to persist. (Confirmed pattern: 10:05 scan wrote escalation_needed: true but did NOT write the issue to issues.jsonl; 12:04 scan had to write it manually.) STALE-PREMISE GUARD (2026-07-14): before writing a gap issue, VERIFY THE LIVE PREMISE is still true — re-scan jobs.json, disk usage, and provider state. A journal flagged escalation_needed: true can carry a premise that resolved AFTER the journal was written; persisting it creates a FALSE escalation (pollutes issues.jsonl and burns an execution-loop cycle). Concrete checks: for oc_state_db_oversized re-derive disk% live (shutil.disk_usage('/root')) — threshold is db>1GB AND disk>80%; if disk is now <80% (even at 5-10GB db) it is acceptable operational cost, do NOT persist. PATH TRAP (confirmed 2026-07-25): the canonical state.db is $HERMES_HOME/../<profile>/state.db (14.7GB in production) — the root `/.hermes/state.db is a 38-byte placeholder and will make you wrongly conclude the oversized premise is FALSE. Measure the PROFILE path (ls -la $HERMES_HOME/..//state.dbplusfind ~/.hermes -name '*.db' -size +1Gto locate all >1GB DBs) before applying the stale-premise guard. A scan that checks only the root path almost failed to persist a genuine 84.9%-disk / 14.7GB-DB oversized condition. For any*_access_token_missing/ auth fingerprint, if the implicated job isstatus=okwith clearedlast_error, it recovered — do NOT persist. For every fingerprint, require ≥1 live job still matching the signature before writing. Confirmed 2026-07-14: oc_state_db_oversized(journal claimed disk 82%) was NOT persisted because live disk was 70.2%;oc_google_tasks_access_token_missingwas NOT persisted becausemonitor:listwas alreadystatus=ok. See references/journal-escalation-stale-premise-guard-2026-07-14.md`.
• 8b-variant — 'tracked' in narrative but escalation_needed: false: A scan may reference root-cause fingerprints as 'tracked' (in previous_scan_delta.stable_root_cause or prose) while setting escalation_needed: false on the journal. If the referenced fingerprint is absent from issues.jsonl, the escalation silently dropped — every later scan re-reports it as 'tracked' without ever persisting it. Fix: after classifying non-auto-fixable root causes, collect their intended fingerprints and verify each exists in issues.jsonl (use scripts/parse_issues_jsonl.py); if missing, WRITE it (status: open, escalation_needed: true). One issue per root-cause fingerprint; list affected job names in affected_components. Confirmed 2026-07-07: 71-job 402-credits and 2-job OAuth revocations were reported 'tracked' in prior deltas but absent from issues.jsonl; light scan wrote them. See references/escalation-persistence-gap.md.
• 8b/8e parser gotcha (confirmed 2026-07-14): When reading issues.jsonl for gap checks, a hand-written brace-walk parser inside python3 << 'PYEOF' that does quote/backslash tracking returns 0 objects (its escape handling breaks on \"). Use json.JSONDecoder().raw_decode instead — robust to escaped quotes and concat-per-line objects. In concurrent cron contexts (sibling custodian:light rewrites the file at top of hour) re-verify the file mtime/size across 3 quick reads before trusting a parse; prefer grep -ac for a single targeted check (e.g. does this issue_id already exist) to avoid a full race-prone rewrite. Race-safe recipe in references/monitor-list-exit1-mask-gap.md.
• 8f. Recurrence-of-fingerprint-CLASS across distinct jobs (gap Step 8b/8b-variant does NOT cover): The journal→issues gap checks only fire when a journal flagged escalation_needed: true but no open issue exists. They do NOT catch a live error job whose fingerprint class was seen before but whose prior issue was correctly resolved per-job (not per-fingerprint-family). Consequence: a same-class error silently recurs on a NEW job with no open issue and no flag, and every scan re-classifies it as "pre-classified" / "known pattern" without persisting anything. Procedure: for each live error job (Steps 6–8), map its error to a reusable fingerprint class (e.g. content_policy_blocked, token_expired, 402 credits, interpreter shutdown, Script exited with code 1). Grep the full issues.jsonl (resolved + open) for that class token. If the class was previously seen but ALL matching issues are now status: resolved/duplicate AND the current job is not listed in any of those issues' affected_job_ids → this is a NEW occurrence of a recurring class on a distinct job → persist a fresh issue (status: open, escalation_needed: true, user_gated per class nature, affected_job_ids = [current job id]) even though no journal flagged it. Do NOT re-open the old resolved issue (it was correct for its own job); write a new dated issue. STALE-PREMISE GUARD still applies: require ≥1 live enabled job currently matching the signature before writing. Confirmed 2026-07-23: ocas-autobio-observe errored live with content_policy_blocked; prior same-class issues oc_bones_content_policy_blocked_20260720 and oc_sands_evening_brief_content_policy_20260722 were both resolved (Mentor note "re-escalate if recurrence detected") and listed only their own jobs — no open issue covered ocas-autobio-observe, so a new issue oc_autobio_content_policy_blocked_20260723T0505Z was written. See references/recurrence-fingerprint-class-distinct-job-2026-07-23.md.
• 8c. Verify-before-accepting-self-resolved: when a prior scan classified an error as "self-resolved" (e.g., ModuleNotFoundError that supposedly fixed itself), verify by running the actual import in the cron execution python — NOT any assumed venv path. Cron jobs run python3 from PATH. To find the actual python: which python3 in a terminal, then python3 -c "import <module>". The profile venv path ($HERMES_HOME/../<profile>/venv/bin/python3) may NOT exist — the system hermes venv (<hermes-venv>/bin/python3) is typically the active one. Confirmed 2026-07-01: dispatch:triage-morning was classified "self-resolved" but the verification was done by checking the actual import (import google.oauth2.credentials → OK). Do NOT accept "self-resolved" from a prior journal entry without re-verifying when: (a) the module path in the error differs from what you assumed, (b) the error was from a no_agent script (different python resolution), or (c) the prior scan has known counting discrepancies. See references/self-resolved-module-verification-pattern.md.
• 8d-variant — 503 scope-expansion gap: If oc_provider_503_upstream_capacity exists but covers fewer live 503 jobs than erroring, record the gap in the journal under scope_gaps and flag for Mentor to expand affected_job_ids. Do NOT write a duplicate issue — one fingerprint = one issue. See references/503-scope-expansion-2026-07-28.md.
• 8e. Verify resolved CODE-DEFECT fixes actually cover all references (catches what reopen_false_resolutions.py misses): that script only matches provider/auth/credit outage signatures (token_expired, 402 credits, owl-alpha 404) — it does NOT catch a resolved issue whose fingerprint is a code defect (oc_*_bug, oc_*_missing, oc_*_path_*) whose fix was incomplete. For each such resolved issue: (a) does any enabled job's last_error STILL contain the original error signature? If yes, first compare the job's last_run_at against the fix timestamp — if the job ran BEFORE the fix landed, the error is STALE (re-run the script to confirm), not a live regression; (b) does the source file the fix touched STILL contain the broken reference the error named? A "resolved" entry whose fix only added a comment saying 'we now use X' while the erroring line still references old Y is a FALSE resolution — reopen (status: open, escalation_needed: true, tier: 4, clear resolved_at, set reopened_at + reopen_note). Grep recipe: grep -rn "<broken_token>" <file>. Re-run the script directly (<hermes-venv>/bin/python <script> or bash <wrapper>) and inspect exit code + stderr to separate stale from live. CORRECTED 2026-07-13 (supersedes the original misread assertion): The original reopen directive for oc_chronicle_facts_fts_missing_20260713 was a MISREAD and must NOT be followed. It claimed enrich_embeddings.py:121 still executed DELETE FROM facts_fts, but line 121 is a COMMENT; the executable statements (lines 130/149) use the live belief_fts/observed_fts tables, which exist in chronicle.db and rebuild cleanly (verified by dry-executing both DELETEs against the live DB — no OperationalError). The job's stored error was STALE: last ran 2026-07-13 10:02 UTC, BEFORE the fix landed (file mtime 11:36 UTC). The resolution was CORRECT — leave it resolved. Grep-pitfall (root cause of the misread): grep -rn "facts_fts" enrich_embeddings.py matches only the historical comment lines (120, 124), producing a false "still broken" hit. A token appearing in comments is NOT proof the broken code path runs — read the actual executable lines around the cited line number, or execute the real code path, before reopening. Always compare last_run_at vs the fix file's mtime: an error from a run BEFORE the fix is stale, not a live regression. FALSE-CLOSE via drained-backlog test (inverse, confirmed 2026-07-13): When a prior scan resolved a timeout/volume code-defect issue on the strength of a live re-run that completed fast, verify that re-run executed against the REAL production backlog — NOT a queue just drained by a prior run. A timeout issue can finish in ~163s when there is nothing left to embed, then HANG past the 600s cron hard limit once daily volume rebuilds. In this session oc_script_timeout_chronicle_embed_20260713 was closed at 21:36Z citing a 163s pass, but its embed_state.json showed last_run=21:35:56 (a prior run had just drained the queue), and a fresh re-run at 23:0xZ was still actively embedding at 85s (row 4160/8000 of the facts pass; facts table holds 35,486 rows) — confirming the timeout is recurring. Before accepting a timeout/throughput 'resolved': (1) check the progress/state file's last_run vs the claimed fix time — if within seconds, the test ran on a cleared queue; (2) inspect real data volume (SELECT COUNT(*) FROM facts); (3) re-run the actual script against that full volume with a hard cap (background terminal(background=true, notify_on_complete=true) + process(wait/poll) — foreground cap is 60s) and confirm it completes under the cron limit. A clean run immediately after another successful run proves nothing about steady-state load. See references/resolved-codefix-regression-verify.md and references/resolved-timeout-verify-drained-backlog.md.
• 9. Verify-before-acting: for any error job, check current config.yaml and provider state to confirm the error is still active before attempting fix
• 10. Write observation journal (even if no issues found — set not_activity_reason)
• 10b. LLM-necessity guard: Run scripts/classify_llm_necessity_integration.py to detect new cron jobs whose prompts don't need LLM reasoning (script-wrappers, self-updates, needless skill-load). This uses the acknowledgment file (data/llm_necessity_ack.json) to avoid re-reporting already-triaged candidates. Writes/updates one oc_cron_llm_unnecessary issue in issues.jsonl for new unacknowledged candidates. REPORT-ONLY — never auto-converts a job to no_agent. Run via: python3 <hermes-home>/profiles/indigo/skills/ocas-custodian/scripts/classify_llm_necessity_integration.py.

Cron silence protocol: When running as a scheduled cron job, if the scan finds no actionable issues, respond with exactly [SILENT]. Only produce a report when there is genuinely new information.

Journal-before-silent requirement: The recovery contract (see spec-ocas-recovery.md) requires every scheduled run to write an evidence record. Even a no-op scan with no actionable issues MUST write an observation journal (with not_activity_reason set) before returning [SILENT]. The correct sequence is: (1) write the journal → (2) return [SILENT]. Do NOT skip the journal on silent runs. The journal proves the scan ran; [SILENT] prevents unnecessary delivery noise.

Deep Scan (optimized 6h cron): Full 13-step sweep. See references/deep-scan.md and references/deep-scan-2026-06-28-clean-verdict.md for the clean verdict pattern (all-transient → journal + silent).

Deep Scan early-exit shortcut: When all error jobs are transient (cf=0/None, last_run before recent restart, no new fingerprints, no consecutive_failures >= 1), skip Steps 3b/4/5/9. Go directly to classification + Tier 1 fix pass. Trigger: all jobs have consecutive_failures in (0, None) AND all last_error match transient patterns (futures shutdown, exit 1 no-op, gateway collision, 429, script-not-found race, no_agent path mismatch, gateway restart import window, provider error transient). Do NOT skip journal or conformance checks. See references/deep-scan.md.

Deep Scan clean verdict (2026-06-23): When ALL error jobs classify as transient/non-faulty (known patterns, cf=None/0, no active issue), the scan is clean. No Tier 1 fixes needed. Write observation journal with not_activity_reason and return [SILENT]. This is the expected steady-state — a clean scan means the system is healthy, not that the scan missed something. Do not force-fix non-issues. Confirmed 2026-06-23: 16 error jobs, 100% transient (futures shutdown, 429 rate limit, no_agent exit 1 noop), 0 fixes applied, all clean.

Delta journal for repeated clean verdicts: When consecutive scans find the SAME errors with no new/resolved issues, journal a previous_scan_delta block: elapsed min, new_issues=0, new_errors=0, stable root cause. New transient errors alongside stable escalated issues: MAY still use delta but include new_errors: N + new_error_detail. Do NOT use after gateway restart, fix, or state change. See references/light-scan-2026-06-29-0904.md.

• Fix-loop already escalated — don't re-escalate: If fix-loop RCA Pattern B exists AND prior esc-run already escalated → note fingerprint in journal, do NOT re-fix, do NOT duplicate escalation, return [SILENT] with not_activity_reason: "clean_verdict_all_errors_already_escalated". See references/deep-scan-fix-loop-prehandled-silent-verdict.md.

Config empty section: "Tier 1 auto-fix" vs Pattern B contradiction: oc_config_empty_section is Tier 1 auto-fix BUT has Pattern B RCA. Resolution: DO remove null keys (fixes TUI warnings), note fix-loop in journal, write escalation for architectural root cause (gateway regenerates null keys on restart). See references/config-empty-section-fixloop-status.md.

Escalation Runner Checklist:

• 1. Check latest esc-run journal first (5-sec check vs 60-sec full scan)
• 1b. Already-classified fast path: If the prior esc-run journal is < 2h old AND classified all open issues as open_user_gated AND no new last_error messages appear in jobs.json that weren't in the prior journal AND no new entries appear in any issues.jsonl → write journal referencing the prior classification and return [S Skip Steps 2-6. See references/escalation-runner-already-classified-fast-path.md`. Do NOT use after gateway restart, after applying a fix, or when a significant state change occurred.
• 2. Discover ALL issues.jsonl paths: find <hermes-home> -name "issues.jsonl"
• 3. Deduplicate by issue_id/id — keep best status per entry
• 4. For each open issue, verify against raw file (terminal(command="cat ...")) — not custodian_issues tool (stale cache)
• 5. Classify into four buckets: Actionable / User-gated / Legacy-inactive / Already-resolved
• 6. If any Actionable issues exist → execute fixes
• 7. If no Actionable issues → write journal with not_activity_reason, return [SILENT]
• 8. Clear escalation_needed flag on any resolved entries

Escalation runner journal write — use Python always: When running escalation-runner in cron context, write journals via python3 -c "..." with json.dump() and from datetime import datetime, timezone; datetime.now(timezone.utc). Do NOT use cat > file << 'EOF' heredoc for JSON containing timestamps/run_ids — single-quoted heredoc prevents $(date) expansion, producing corrupted files. Import the CLASS, not the module: import datetime; datetime.now() raises AttributeError: module 'datetime' has no attribute 'now'. Always use from datetime import datetime, timezone so datetime.now(timezone.utc) resolves to the class method. See references/escalation-runner-already-classified-fast-path.md § Journal write pattern reminder.

The fos→os typo in custodian scripts (confirmed 2026-07-27, NOW FIXED): fos.path.expanduser (should be os.path.expanduser) appeared in 8 files across the custodian scripts/ directory — a copy-paste error from an earlier module. Affected files: classify_error_jobs.py (lines 20, 90), confirm_provider_recovery.py (134), verify_provider_recovery.py (24, 40), verify_escalation_state.py (161), find_missed_user_gated_jobs.py (187), bucket_error_jobs.py (138), classify_llm_necessity.py (27), classify_llm_necessity_integration.py (24, 25, 273). Impact: classify_error_jobs.py crashes block Step 6 de-aggregation of exit-1 wrapper jobs; classify_llm_necessity_integration.py crashes block Step 10b LLM-necessity integration. Fix: s/fos\.path\.expanduser/os.path.expanduser/g across all files. If one occurrence is fixed, check ALL files in the skill's scripts/ directory — it is a copy-paste error that recurs. See references/fos-nameerror-pattern.md for full details and impact analysis. Resolved 2026-07-27: all 8 files patched, verified 0 remaining occurrences.

Escalation runner clean verdict — actionable vs user-gated vs legacy (2026-06-25): When the escalation runner finds no actionable issues, classify all open entries into four buckets: (A) Actionable — execute fix; (B) User-gated — note count but do not auto-fix (skill library hygiene, stub removal); (C) Legacy/inactive — ignore YAML debris in profiles with no cron/jobs.json; (D) Already resolved — verify config/job state and close. If Bucket A is empty, write journal with not_activity_reason and return [SILENT]. See references/escalation-runner-clean-verdict-pattern.md for the decision tree and journal template. Inactive profile detection: check for cron/jobs.json absence (>90 days dormant). Confirmed 2026-06-25: braun profile has 3 null keys but no cron jobs — legacy debris, not an action item.

Escalation runner: user-gated provider failures are not permanent kill switches (2026-07-09): Do NOT pause recurring cron jobs just because their last error is provider auth/credits/429/endpoint outage. First verify the provider/model live with a minimal hermes chat -q probe. If the provider works now, resume affected jobs and let them retry; stale last_error is not live failure. Only pause when retry is genuinely futile until user action (e.g. revoked Google OAuth for a domain-specific tool, missing script, blocked execute_code redesign), and then write pause_reason, jobs_paused, journal evidence, and a re-enable-on-recovery check. Pausing is mitigation, never resolution. Detailed procedure: references/stale-provider-error-pause-loops.md.

Escalation Execution Loop (external cron trigger) — EXECUTE, don't just classify (2026-07-07): When an external loop invokes Custodian + Mentor to execute fixes on escalated issues (not merely classify them), the default Escalation Runner Checklist above is classification-oriented — this fills the execute-and-reconcile gap:

See references/escalation-loop-pitfalls.md for chronic traps in execution-loop runs: the journal-gap probe reports FALSE gaps for already-resolved issues (it compares only against OPEN issues), stale last_error vs live failure (inspect the wrapper/script + fix timestamp before reopening), cooperating with an in-flight sanctioned sibling repair instead of racing it, and reading the traceback location (not just the final exception) when a database is locked error appears after the root-cause step already passed.

1. Verify live state BOTH directions against jobs.json — (a) issue claims resolved/paused but job still enabled+erroring (inverse gotcha), AND (b) issue flags escalation_needed: true but the job already recovered (last_status: ok, last_error cleared). Resolve or re-pause accordingly; never trust the issue flag alone. AND (c) sweep for missed enrollments: any enabled+erroring job whose last_error matches a known user-gated fingerprint (Nous 401 portal.nousresearch.com, OpenRouter 402 credits, owl-alpha 404, Google Tasks 403, invalid_grant) but is NOT in any issue's jobs_paused. These failed in the inter-scan window after the last esc pass and were never enrolled. For provider/model fingerprints, verify the provider live first; if it now works, leave/resume the jobs and clear stale issue state

…(truncated)