Hindsight Maintenance

Hindsight Maintenance

Keep the self-hosted Hindsight memory system healthy across every bank on a schedule. The expensive, dangerous, and repetitive parts — bank enumeration, the health sweep, the orphaned-job reaper, the new-memory watermark — are pre-paid in scripts/. Spend agent judgment only on the creative optimization passes (pollination, synthesis, prune review), never on re-deriving the CLI or SQL surface.

Endpoint https://api.hs.delo.sh. The database is host-native Postgres at 127.0.0.1:5432, database hindsight, schema hindsight (not public) — this custom schema is the root of two upstream bugs the tune-up workflow guards against. Stack and DB repair SQL live in ~/docker/stacks/ai/hindsight/ (compose.yml, .env, fixes/).

Operating Principles

• Read the evidence before touching anything. Every workflow opens with scripts/tune-up.sh output or a hindsight memory recall, never ad-hoc psql/docker logs spelunking.
• Destructive ops are dry-run first, user-gated second. memory delete, clear-observations, and pruning are irreversible; memories are expensive to rebuild. Show the candidate set, get a decision, then act.
• One bank's failure never aborts the sweep. Iterate all banks, collect errors, keep going.
• Never hand-roll a cross-worker DB write. Reaping stale jobs is subtle — always use scripts/reap-orphans.sh, never a hand-written UPDATE (see Cross-cutting rules for why).
• Watermarks gate "new since last run." State lives in ~/.hindsight/maintenance/, advanced only after the consuming step (wiki compile) succeeds — a crash re-processes, never skips.

Bank enumeration (every workflow's first move)

SKILL_DIR is the base directory stated when this skill loaded. Set it once per shell (the fallback below is correct on this host) so every "$SKILL_DIR/…" command in the references resolves:

export SKILL_DIR="${SKILL_DIR:-$HOME/.claude/skills/hindsight-maintenance}"
source "$SKILL_DIR/scripts/hs-lib.sh"   # HS_STACK_DIR, hs_db, hs_banks, hs_live_worker
hs_banks            # every bank_id, one per line (≈141)

Cadence → workflow

Intent → reference

Workflow families (quick pointers; open the reference before running)

• tune-up — scripts/tune-up.sh writes a read-only health report: global queue state, orphaned jobs (dead-worker claims), missing public.banks_needing_consolidation() / public.schemas_with_expired_rows(), failed ops, and per-bank anomalies. Reap with scripts/reap-orphans.sh --apply; repair missing functions from ~/docker/stacks/ai/hindsight/fixes/.

• optimization — four passes, most-destructive last: consolidate (hindsight bank consolidate <bank> — synthesize observations), prune (low access_count + age + duplicate heuristics → memory delete, dry-run→gate), cross-bank pollination (a theme learned in one bank surfaced into sibling banks it also serves), novel synthesis (reflect over memory combinations, retain emergent observations no single memory held).

• karpathy-wiki — scripts/new-memories.sh <bank> pulls memories retained since the watermark; hand them to the karpathy-llm-wiki skill's Ingest; advance the watermark on success.

Cross-cutting rules

• Schema is hindsight. All SQL is schema-qualified (hindsight.async_operations, hindsight.memory_units). Maintenance functions live in public.* and are shared across schemas — a fresh DB or image upgrade can leave them missing; tune-up checks this every run.
• Reaping requires the live worker to be up. Run the container, let it claim its work, then reap — the reaper protects worker_id = <live hostname> claims. Reaping against a stopped container is unsafe.
• Consolidation is a queued async job, not synchronous. hindsight bank consolidate <bank> enqueues; use --wait to block, or check hindsight operation list <bank>.
• Attribute retained memories. Synthesis/pollination writes go back with --context conventions|architecture and a --doc-id so re-runs upsert instead of duplicating.
• Persist findings. Non-trivial maintenance outcomes get hindsight memory retain infra … (the homelab bank) so the next sweep starts informed.

Out of Scope

• One-off recall/retain during normal task work → use the hindsight skill. This skill is scheduled/bulk upkeep, not the day-to-day memory API.
• Infra-level failure of the Hindsight container or host (crash loops, OOM, disk, Traefik, the container won't start) → use the server-maintenance skill. This skill assumes the service is up and reachable.
• Debugging Hindsight application source / migrations → that is upstream code work in the vectorize-io/hindsight image, not memory upkeep. tune-up only detects and works around those bugs (missing functions, orphaned jobs); it does not patch the image.
• Any non-Hindsight datastore (Postgres for other stacks, Qdrant, Redis) → not this skill.