Custodian Maintenance Runbook
Operates a client's self-hosted stack — booking engine (Cal.diy), automation
(n8n), chat (Typebot), Postgres, web/TLS, Stripe webhook. **Purpose: availability
- recoverability + currency, at the lowest possible inference cost.**
When to Use This Skill
Use this skill when:
- A customer stack must be monitored, backed up, and updated automatically.
- You need a repeatable, guardrailed maintenance loop (health → backup →
update → rollback) that never breaks a live booking flow.
- Cost matters: the loop should run decisions on a cheap model and only summon
a generation model for genuine language work.
COST DISCIPLINE (first rule)
- Use JEV (
typesafe/jev-1.13, endpoint /api/alpha/decisions) for EVERY decision
(gate an update, classify an incident, pick retention).
- Use a cheap flash model ONLY for unavoidable language work: writing the human
incident text, parsing an error JEV can't classify, composing an escalation, reading
a release note for breaking-change judgment.
- A cycle that completes JEV-only is a WIN. Do not summon a generation model for
anything a command, grep, or JEV resolves.
JEV call shape (verified)
POST https://openrouter.ai/api/alpha/decisions
Headers: Authorization: Bearer $OPENROUTER_API_KEY, Content-Type: application/json
Body:
{"model":"typesafe/jev-1.13",
"state":"<short context>",
"questions":{"q1":{"type":"choice","instructions":"...","criteria":{"A":"...","B":"..."}},
"q2":{"type":"score","instructions":"...","criteria":["0: ...","1: ...","2: ..."]},
"q3":{"type":"noul","instructions":"..."}}}
Read answers[qid].choice|score|noul + .confidence. Threshold < 0.6 confidence → escalate.
score runs 0..(len(criteria)-1); normalise / (len-1). Use noul, not bool. Batch all
questions for a decision into ONE call. try/catch every JEV call → safe neutral default.
1. HEALTH PROBE (every 15 min)
Probe each service; record ok/fail:
systemctl is-active <cal-diy|n8n|typebot|postgres|caddy/nginx> 2>/dev/null
curl -fsS -o /dev/null -w "%{http_code}" http://localhost:PORT/health # per service
curl -fsS -o /dev/null -w "%{http_code}" https://<domain>/ # web + TLS via HTTPS
openssl x509 -noout -enddate -in <crt> 2>/dev/null # TLS expiry
df -h / | tail -1; free -m | awk 'NR==2{print $3"/"$2"MB"}' # disk + RAM
If all ok → emit nothing (quiet). If any fail:
- JEV
noul "Is this an incident requiring action?" (threshold 0.6).
- JEV
choice classify: transient / service-down / disk-full / tls-expiring / unknown.
- If covered by a rule (one known-good service dead, TLS < 7d) → repair per that
service's rollback+restart rule; verify health after.
- If unknown/breaking/ambiguous → do NOT mutate → queue, alert once.
2. WEEKLY BACKUP (Sunday 03:00 UTC)
pg_dump the DB to a timestamped file.
- Back up config + env (vault labels, docker-compose, Caddyfile).
- Rotate by retention: daily×7 + weekly×4 (from config).
- Copy off-box to the backup destination (S3/syncthing/second VPS).
- VERIFY restorable (
pg_restore --list or load into scratch schema + SELECT count).
- Alert:
Backup <ts> OK (n MB, verified restorable) or BACKUP FAILED: <why>.
- Never run a mutation without a fresh verified backup first.
3. WEEKLY UPDATE CHECK (Monday 02:00 UTC)
- Query upstream releases: Cal.diy (
calcom/cal.diy), Typebot (baptisteArno/typebot.io),
n8n (n8n-io/n8n). Note current pinned vs available.
- For each pending update: JEV
noul "Is this a safe non-breaking update?"
(fresh backup + supported path + not a major jump = lower; breaking/major/unknown = higher).
- Non-breaking → apply → verify service health → record.
- Breaking/major/ambiguous → STALL, escalate with the specific release + JEV reasoning.
Do NOT auto-install a breaking change on a live booking system.
- Report briefly: applied / stalled / nothing.
- Core stays upstream-compatible: customisation is additive overlay; never a destructive
rewrite that breaks a clean upstream merge.
4. DISK HEADROOM (daily)
df -h /System/Volumes/Data 2>/dev/null || df -h / — take the DATA volume. If < 15%
free → warn. If < 5% → escalate immediately (a full disk kills cron first).
- Flag transient big-writer patterns from shared hosts if present (WAL/snapshots can
spike a volume and starve unrelated cron).
5. ROLLBACK (always have a story before a mutation)
Keep the previous pinned version/image + its verified backup. To rollback: stop service,
restore previous image, restart, verify health. The rollback path exists BEFORE any apply.
6. REPORTING
- Quiet unless it matters. Speak up on: incident fired, update applied, backup
verified/FAILED, escalation needed, disk low.
- Reports are verified on the output side — never claim success without reading back proof.
- Escalation format: what happened · tradeoff · recommendation · exact decision needed.
PITFALLS
- A full disk kills cron first — check disk first, cheaply, often.
score needs normalising; use noul not bool; confidence-gate before any action.
- Never blind-
latest on a live system; pin and verify.
- Never state an update/backup succeeded without output-side verification.
- Secrets by label from the vault; split-headers so keys never verbatim in output.
- DeepSeek/flash is for language work only — don't summon it to "decide".
1---2name: maintenance3description: >-4 Use this skill when operating a self-hosted customer web stack (Cal.diy /5 Typebot / n8n / Postgres / Stripe) that a maintenance agent must keep6 available, patched, and backed up automatically. JEV-first decisions, sparse7 LLM generation, health probes, weekly backups, JEV-gated updates, rollback.8version: 1.0.09platforms: [hermes, openclaw, claude, codex]10author:11 name: Custodian (AgentForge)12 source: Joerg Peetz / Fluffy — Custodian customer-maintenance agent13license: MIT14risk_tier: L215tags: [maintenance, backup, update, health, ops, self-hosted, jev, runbook]16requires:17 binaries: []18metadata:19 source: https://skillmd.com/skills/jpeetz/maintenance20---2122# Custodian Maintenance Runbook2324Operates a client's self-hosted stack — booking engine (Cal.diy), automation25(n8n), chat (Typebot), Postgres, web/TLS, Stripe webhook. **Purpose: availability26+ recoverability + currency, at the lowest possible inference cost.**2728## When to Use This Skill2930Use this skill when:31- A customer stack must be monitored, backed up, and updated automatically.32- You need a repeatable, guardrailed maintenance loop (health → backup →33 update → rollback) that never breaks a live booking flow.34- Cost matters: the loop should run decisions on a cheap model and only summon35 a generation model for genuine language work.3637## COST DISCIPLINE (first rule)38- Use **JEV** (`typesafe/jev-1.13`, endpoint `/api/alpha/decisions`) for EVERY decision39 (gate an update, classify an incident, pick retention).40- Use a cheap **flash model** ONLY for unavoidable language work: writing the human41 incident text, parsing an error JEV can't classify, composing an escalation, reading42 a release note for breaking-change judgment.43- A cycle that completes JEV-only is a WIN. Do not summon a generation model for44 anything a command, grep, or JEV resolves.4546## JEV call shape (verified)47POST `https://openrouter.ai/api/alpha/decisions`48Headers: `Authorization: Bearer $OPENROUTER_API_KEY`, `Content-Type: application/json`49Body:50```json51{"model":"typesafe/jev-1.13",52 "state":"<short context>",53 "questions":{"q1":{"type":"choice","instructions":"...","criteria":{"A":"...","B":"..."}},54 "q2":{"type":"score","instructions":"...","criteria":["0: ...","1: ...","2: ..."]},55 "q3":{"type":"noul","instructions":"..."}}}56```57Read `answers[qid].choice|score|noul` + `.confidence`. Threshold `< 0.6` confidence → escalate.58`score` runs 0..(len(criteria)-1); normalise / (len-1). Use `noul`, not `bool`. Batch all59questions for a decision into ONE call. try/catch every JEV call → safe neutral default.6061## 1. HEALTH PROBE (every 15 min)62Probe each service; record ok/fail:63```64systemctl is-active <cal-diy|n8n|typebot|postgres|caddy/nginx> 2>/dev/null65curl -fsS -o /dev/null -w "%{http_code}" http://localhost:PORT/health # per service66curl -fsS -o /dev/null -w "%{http_code}" https://<domain>/ # web + TLS via HTTPS67openssl x509 -noout -enddate -in <crt> 2>/dev/null # TLS expiry68df -h / | tail -1; free -m | awk 'NR==2{print $3"/"$2"MB"}' # disk + RAM69```70If all ok → emit nothing (quiet). If any fail:711. JEV `noul` "Is this an incident requiring action?" (threshold 0.6).722. JEV `choice` classify: transient / service-down / disk-full / tls-expiring / unknown.733. If covered by a rule (one known-good service dead, TLS < 7d) → repair per that74 service's rollback+restart rule; verify health after.754. If unknown/breaking/ambiguous → do NOT mutate → queue, alert once.7677## 2. WEEKLY BACKUP (Sunday 03:00 UTC)781. `pg_dump` the DB to a timestamped file.792. Back up config + env (vault labels, docker-compose, Caddyfile).803. Rotate by retention: daily×7 + weekly×4 (from config).814. Copy off-box to the backup destination (S3/syncthing/second VPS).825. VERIFY restorable (`pg_restore --list` or load into scratch schema + SELECT count).836. Alert: `Backup <ts> OK (n MB, verified restorable)` or `BACKUP FAILED: <why>`.84- Never run a mutation without a fresh verified backup first.8586## 3. WEEKLY UPDATE CHECK (Monday 02:00 UTC)871. Query upstream releases: Cal.diy (`calcom/cal.diy`), Typebot (`baptisteArno/typebot.io`),88 n8n (`n8n-io/n8n`). Note current pinned vs available.892. For each pending update: JEV `noul` "Is this a safe non-breaking update?"90 (fresh backup + supported path + not a major jump = lower; breaking/major/unknown = higher).913. Non-breaking → apply → verify service health → record.924. Breaking/major/ambiguous → STALL, escalate with the specific release + JEV reasoning.93 Do NOT auto-install a breaking change on a live booking system.945. Report briefly: applied / stalled / nothing.95- Core stays upstream-compatible: customisation is additive overlay; never a destructive96 rewrite that breaks a clean upstream merge.9798## 4. DISK HEADROOM (daily)99- `df -h /System/Volumes/Data 2>/dev/null || df -h /` — take the DATA volume. If < 15%100 free → warn. If < 5% → escalate immediately (a full disk kills cron first).101- Flag transient big-writer patterns from shared hosts if present (WAL/snapshots can102 spike a volume and starve unrelated cron).103104## 5. ROLLBACK (always have a story before a mutation)105Keep the previous pinned version/image + its verified backup. To rollback: stop service,106restore previous image, restart, verify health. The rollback path exists BEFORE any apply.107108## 6. REPORTING109- Quiet unless it matters. Speak up on: incident fired, update applied, backup110 verified/FAILED, escalation needed, disk low.111- Reports are verified on the output side — never claim success without reading back proof.112- Escalation format: what happened · tradeoff · recommendation · exact decision needed.113114## PITFALLS115- A full disk kills cron first — check disk first, cheaply, often.116- `score` needs normalising; use `noul` not `bool`; confidence-gate before any action.117- Never blind-`latest` on a live system; pin and verify.118- Never state an update/backup succeeded without output-side verification.119- Secrets by label from the vault; split-headers so keys never verbatim in output.120- DeepSeek/flash is for language work only — don't summon it to "decide".