Hermes Cron Ops

Hermes Cron Job Ops

Operate, diagnose, and repair Hermes scheduled cron jobs (the cronjob tool). This skill covers the failure modes that are NOT visible from the cron registry's last_status field — the cases where a job reports success but did nothing, or aborts with a spend-guard error before doing any work.

When to use

• A cron job shows last_status: error and you need the root cause.
• A cron job reports ok but produced no real output (false positive).
• You changed the global inference provider/model and jobs stop running.
• A cron run logs Skill(s) not found and skipped.
• You need to pin or re-pin a job's model/provider after a config change.

Diagnostic workflow (always in this order)

1. List, then READ the output logs. cronjob action=list gives status, but the real story is in ~/.hermes/profiles/<profile>/cron/output/<job_id>/<timestamp>.md. Read the most recent file. Two signatures tell you everything (see references/cron-diagnostics.md):

   • RuntimeError: Skipped to prevent unintended spend: global inference config drifted → drift guard (Failure mode A).
   • [IMPORTANT: The following skill(s) were listed for this job but could not be found and skipped: ...] → skill not loading (Failure mode B).

2. Inspect the backing store directly. ~/.hermes/profiles/<profile>/cron/jobs.json holds the truth: "provider", "model" (explicit pin — often null), "model_snapshot" (the config captured at creation; this is what the drift guard compares against), "last_error". The API list/update responses under-report (see Pitfalls), so read this file to confirm ground truth rather than trusting the tool's echo.

3. Verify a fix with a live run + output grep — never by last_status alone. last_status: ok is NOT proof of success. A job whose skills were skipped still reports ok while emitting garbage (e.g. a degenerate \boxed{A} answer instead of the real task). After any repair, run cronjob action=run job_id=<id> then check the NEW output file:

   f=$(ls -t ~/.hermes/profiles/<profile>/cron/output/<job_id> | head -1)
   grep -q "Skipped to prevent unintended spend" "$f" && echo "GUARD ERROR"
   grep -q "Skill(s) not found and skipped"        "$f" && echo "SKILL SKIP"
   

   Absence of BOTH = genuinely healthy. Also confirm the run actually loaded the skill (e.g. The user has invoked the "X" skill… The full skill content is loaded below).

Failure mode A — Inference-config drift guard (hard abort)

Symptom: run fails instantly with RuntimeError: Skipped to prevent unintended spend: global inference config drifted since this job was created (model '<old>' -> '<new>'), and this job is unpinned. No inference call was made.

Cause: the job was created unpinned ("model": null). After creation, the global default inference config (provider and/or model) changed. Hermes refuses to spend on an unpinned job whose effective config no longer matches what it was created with. There are two axes: model drift (stepfun/... -> tencent/hy3:free) and provider drift (nous -> custom).

Fix — pin the job to the current config. Use cronjob action=update with BOTH provider and model set:

cronjob action=update job_id=<id> provider=<prov> model=<model>

The guard clears once model_snapshot (in jobs.json) equals the current global model, and provider is explicitly set. Confirm by reading jobs.json, not the API echo.

Quirk — model-only updates are silently dropped. Sending cronjob action=update job_id=<id> model=<model> with no provider returns {"error":"No updates provided."} and changes nothing. The explicit model pin may stay null even when the call "succeeds" while paired with provider. ALWAYS send provider and model together.

Failure mode B — Skills silently skipped (soft failure, false ok)

Symptom: output opens with [IMPORTANT: The following skill(s) were listed for this job but could not be found and skipped: research-pipeline, llm-wiki, arxiv]. The job may still report last_status: ok because inline-prompt execution didn't throw — it just ran without the skill's instructions.

Cause (most common): the skill is installed in a nested duplicate directory, e.g. BOTH skills/research/llm-wiki/SKILL.md AND skills/research/llm-wiki/llm-wiki/SKILL.md. The loader then sees two matches and either (a) throws Ambiguous skill name: 2 skills match… Refusing to guess when resolved via skill_view, or (b) silently skips the skill during cron load. The nested copy is almost always a stale duplicate (verify with md5/diff before deleting). See hermes-agent-skill-authoring for the canonical one-SKILL.md-per-dir layout.

Fix: remove the nested duplicate dir, keeping the canonical (outer) copy which also holds references/ and scripts/:

cd ~/.hermes/profiles/<profile>/skills/research
rm -rf llm-wiki/llm-wiki arxiv/arxiv research-pipeline/research-pipeline

Before deleting, confirm the outer copy is complete and the nested copy adds nothing: diff the two SKILL.md files, and check the outer dir for references/ + scripts/ the nested one lacks. If the two copies DIVERGE (different checksums), decide which is canonical before deleting either.

Other causes of skill-skip: WIKI_PATH/env mismatch, or the skill simply not installed for that profile. Rule those out if no nested dup exists.

Verification checklist (end of any cron repair)

• jobs.json shows provider = current global provider AND model_snapshot = current global model (drift cleared).
• No nested duplicate skill dirs remain.
• Live cronjob action=run produces an output file with NEITHER guard error NOR skill-skip notice.
• Output file shows the skill actually loaded and did real work (not a degenerate stub).

Pitfalls

• Never trust last_status: ok as success-proof. Always read the output file. In a real incident a Monday job reported ok while emitting \boxed{A} because its skills were skipped.
• cronjob update with a model-only payload is a no-op — always pair model with provider.
• The API list/update responses under-report. model can show null post-update while model_snapshot (in jobs.json) actually changed. Read jobs.json for ground truth.
• macOS HFS+ hides case differences (llm-wiki vs LLM-Wiki both resolve) — that is NOT the cause of skill-skip when a genuine nested-dup structure exists. Don't chase case when the dir tree shows real nesting; fix the nesting instead.

References

• references/cron-diagnostics.md — exact error signatures, jobs.json field map, and the grep verification recipe, with a worked example from a real incident.