Citadel Proactive Ingest

Citadel Proactive Ingest

Capture durable engineering knowledge into Citadel as it happens, with no per-session ceremony. Five layers — proactive agent reads, dev-side hooks, plus server-side cron — all fail-silent and personal-by-default unless explicitly promoted or volunteered.

0. Task start, agent-driven (proactive read). Before coding on a project question, run citadel_search (Central + your Node + Shared Session Traces). Treat trace hits as reference-only prior work — verify before acting; Central stays org-authoritative. After a hard dead end the user wants teammates to skip, ask for approval then call citadel_share_session.
1. Mid-session, agent-driven. While working, the agent proactively calls citadel_ingest for durable facts and decisions. Personal-by-default.
2. Git push, hook-driven (universal baseline). On every git push (all IDEs), a pre-push hook snapshots commit metadata to the dev's Node — install once per clone.
3. SessionEnd, hook-driven (optional). On every session close (Claude Code), a non-blocking hook distills the transcript and POSTs a short note to the dev's Node — zero per-session steps after a one-time token export.
4. Server-side Railway cron (org-wide). Scheduled jobs sync GitHub org digests, Linear workspace → Central with Seat-Scoped Mirrors, skills catalog refresh, self-improvement, and backup mirror export. Devs never trigger these manually.

Personal node:  seat:{slug}   (the dev's private Citadel node — default target)
Shared Central: masumi-network (org-wide; Promotion Agent + org sync — not seat tags)
Hosted base:    https://citadel.utxo.ag

Setup first (token + MCP): https://citadel.utxo.ag/skills/connect Read/write rules: https://citadel.utxo.ag/skills/vault

Personal-by-default (read first)

A seat-writer token carries default_dataset=seat:{slug}. Send writes with no dataset field and the server's resolve_write_targets routes them to your private Node. Seat-scoped writes never dual-write to Central via tags — ADR-0007 Seat Node Write Policy.

Central updates for your notes happen when the Promotion Agent rules pass (known masumi-org work) or when you approve a New Org Project proposal in the Operations Dashboard, MCP, or citadel promotion CLI.

Layer 0 — proactive search and shared session traces

Search before coding. At the start of a task — architecture, debugging, "how do we do X here" — run citadel_search without waiting to be asked. Default scope includes your Node, Central, and session-traces (Shared Session Traces volunteered by teammates).

Trust demotion on trace hits. Results are split (central, session_traces, node). Every trace hit carries _citadel.trust: reference-only, plus author_seat and age. Traces are consultable prior work — not Structured Knowledge. Never treat them as org truth; verify against code, Central, or the author before acting.

Share dead-end routes (explicit consent only). When the session surfaced a costly wrong turn another teammate would likely repeat, ask the user:

"Want to share this session's route so teammates can find it via search?"

Only after yes, call citadel_share_session (from an Approved Capture Root). It uploads Compact Session Context — distilled recap + dead ends, never raw transcript. SessionEnd hooks still write private Node traces only; sharing is never automatic in v1.

Cron is not for agents. Railway cron syncs GitHub/Linear org sources and runs the learning pipeline — it does not replace agent citadel_search or per-agent share decisions. Do not schedule cron jobs for per-seat agent behavior.

Layer 1 — proactive mid-session ingest (agent behavior)

While working in an org repo, when a durable fact crystallizes, capture it without waiting to be asked — via the headless CLI (citadel ingest), or the citadel_ingest MCP tool when your session has it registered. Keep each note small and curated.

Ingest proactively when:

• A decision is made (approach chosen, tradeoff settled, library picked).
• A non-obvious root cause is found, or a bug class is understood.
• An interface/contract, env var, or config invariant is established.
• A reusable runbook or "how we do X here" emerges.

Personal-by-default call (lands in your node):

# Headless CLI — works in any terminal/runner with CITADEL_MCP_ACCESS_TOKEN set
citadel ingest "Picked urllib over requests for the SessionEnd hook so it stays stdlib-only and dependency-free." \
  --tag dev-session --tag decision

# MCP equivalent (when citadel_* tools are registered in-session)
citadel_ingest(
  data="Picked urllib over requests for the SessionEnd hook so it stays "
       "stdlib-only and dependency-free.",
  tags=["dev-session", "decision"],
)

No dataset field → seat Node. Tag with context (adr, runbook, repo name) for search — tags do not route seat writes to Central.

Never ingest (same rules as citadel-vault): secrets, tokens, keys, seed phrases, PII, raw logs/debug dumps, ephemeral chatter, or large uncurated dumps. Summarize durable decisions and source facts instead.

Install: all hooks are now bundled in the citadel-archive package (kb.hooks.*) and installed by citadel onboard — no vendored scripts. See the citadel-onboard skill. The sections below describe what each hook does and how to wire it by hand.

Layer 2 — SessionEnd auto-sync (the hook)

python -m kb.hooks.sync_session runs from a Claude Code SessionEnd hook. On every session close it:

1. reads the hook payload (transcript_path, cwd, session_id, hook_event_name) from STDIN;
2. parses the transcript JSONL defensively (skips malformed lines, never crashes);
3. distills a short, deterministic note — 1-2 line recap, key decisions, files changed, notable facts — with no local LLM call;
4. derives tags = ["dev-session", <git-branch>, <repo-name>];
5. truncates to CITADEL_MCP_MAX_INGEST_BYTES (default 200000);
6. POSTs {data, tags} (NO dataset → personal node) to {base}/ingest with Authorization: Bearer ${CITADEL_MCP_ACCESS_TOKEN}, HTTPS only;
7. fails silently — catches everything, exits 0, never blocks session close, never prints the token.

Server-side LLM enrichment (CITADEL_LLM_ENRICHMENT_ENABLED) does any deeper structuring after the note lands; the hook stays deliberately dumb and fast.

One-time token setup (the only step)

Get a seat-writer token from the connect wizard (https://citadel.utxo.ag/skills/connect) and export it once:

export CITADEL_MCP_ACCESS_TOKEN='ctdl_...'

That's it. The same token already powers citadel_search and the MCP config, so most devs already have it set. With it in the environment, every session auto-syncs to your private node. With it unset, the hook is a clean no-op.

Wire the hook into a repo

citadel onboard merges a SessionEnd hook into .claude/settings.json whose command is "<python>" -m kb.hooks.sync_session. To do it by hand, add that command under hooks.SessionEnd with allowedEnvVars: ["CITADEL_MCP_ACCESS_TOKEN"].

Layer 3 — Git push auto-sync (universal baseline)

python -m kb.hooks.sync_push runs from a git pre-push hook. On every push it:

1. reads pre-push ref lines from STDIN (or HEAD when invoked manually);
2. collects commit hash, subject (not body), author, branch, and changed file paths — no raw diffs and no commit message body;
3. POSTs {data, tags} with tags = ["git-push", <branch>, <repo>] (NO dataset → personal Node), HTTPS only;
4. fails silently — always exits 0, never blocks git push.

Install: citadel onboard writes a self-contained .git/hooks/pre-push that runs the bundled module. To do it by hand:

printf '#!/bin/sh\n"%s" -m kb.hooks.sync_push "$@" 2>/dev/null || true\nexit 0\n' \
  "$(command -v python3)" > .git/hooks/pre-push
chmod +x .git/hooks/pre-push

Same CITADEL_MCP_ACCESS_TOKEN as SessionEnd and MCP. Works in Cursor, Codex, and Claude — any tool that uses git. This is the only required dev-side sync step for non-Claude IDEs.

Approved Capture Roots (required allowlist)

Push capture is fail-closed. Without ~/.citadel/capture.json (or with an empty/corrupt allowlist), the pre-push hook captures nothing. citadel onboard seeds the current repo as a root; to add more folders:

citadel setup                                   # interactive
citadel setup --root "$HOME/work/repo=org-work" # or non-interactive

Each root carries Capture Root Tags: personal (never promoted to Central) or org-work (eligible for Promotion Agent review). The token is not stored in the file — it stays in the environment.

Once roots exist, sync_push.py only captures pushes from inside an Approved Capture Root (others are skipped with a warning). Capture on demand with:

citadel capture --dry-run   # preview per-root summaries (no network)
citadel capture             # POST git-metadata + README summaries to your Node

Hardening notes: the Node URL must be HTTPS (citadel capture refuses http:// before sending the token, and never follows redirects). Summaries are size-capped (CITADEL_MCP_MAX_INGEST_BYTES, default 200000) and git-remote credentials are redacted. citadel capture exits non-zero on any failure (corrupt config, no matching root, missing token, or a per-root POST error), so it is safe to gate CI on.

Layer 4 — Server-side Railway cron (org-wide)

Background jobs on Railway keep Central and Seat-Scoped Mirrors fresh. Devs and agents never need to trigger these — cron is the source of truth for org-wide source material.

Key env vars (Railway cron service / web service — not dev shells):

• CITADEL_LINEAR_API_KEY — required for linear-sync (and the evolve Linear stage)
• CITADEL_EVOLVE_SCHEDULER_ENABLED / CITADEL_EVOLVE_INTERVAL_SECONDS — the hourly in-process evolve scheduler (web service; default 3600)
• CITADEL_LINEAR_USER_MAP — optional JSON map Linear user id → seat slug
• CITADEL_GITHUB_ORG, CITADEL_GITHUB_SYNC_* — GitHub org sync scope
• CITADEL_PIPELINE_*_ENABLED — toggle individual pipeline stages

Manual admin triggers (only when the user explicitly asks):

• POST /api/github-sync/run or MCP citadel_run_learning_agent
• POST /api/linear-sync/run
• POST /api/learning-agent/optimize

Agent sync policy

Rely on cron + hooks — do not add dev ceremony.

All background sync (hooks and cron) is fail-silent: outages never block git push, session close, or agent work.

Privacy posture

• Token from env only. Read solely from CITADEL_MCP_ACCESS_TOKEN; never printed, echoed, or written to a tracked file.
• Distilled, not raw. The SessionEnd hook sends a curated short note, not the transcript. Push hooks send subject + paths only.
• Allowlist-required. Missing capture.json → push capture is a no-op.
• Transcript path allowlist. SessionEnd refuses paths outside ~/.claude / ~/.cursor / ~/.codex / ~/.agents.
• Personal-by-default. No dataset field on auto-sync → your seat node.
• HTTPS only. The POST refuses any non-https:// base URL.
• Non-blocking. The hook always exits 0; a Citadel outage never delays your session close.

Opt-out

Drop a SessionEnd override into .claude/settings.local.json (gitignored), or simply unset CITADEL_MCP_ACCESS_TOKEN for that shell. With no token the hook does nothing. See README.md.

Reference

• This skill: https://citadel.utxo.ag/skills/proactive-ingest
• Connect wizard: https://citadel.utxo.ag/skills/connect
• Vault read/write rules: https://citadel.utxo.ag/skills/vault
• Dev onboarding: docs/onboarding/citadel-autosync.md
• Teammate one-pager: docs/onboarding/teammate-rollout.md
• Ingest API: POST /ingest with {data, dataset?, tags?, session_id?} (writer token)
• Seat/Node/Central + Seat-Scoped Mirror: ADR-0003, ADR-0004