Sovereign RAG Ops

Sovereign RAG — Operations Skill

You are operating a PROVEN, battle-tested RAG platform. Do not improvise — follow the recipe. The recipe encodes a week of debugging so you don't repeat it.

Ground rules (non-negotiable)

1. Use the project's pinned Python (the one with psycopg2/pgvector/dotenv installed). The system default often lacks these. Check the project's requirements.txt / .tool-versions / README and use that interpreter.
2. Never print secrets. Passwords/tokens live in the project's untracked .env (or secrets vault). Mask anything credential-like (first 3 chars + length only). Never echo a full value into logs, chat, or files.
3. Shell inline scripts break on quotes/asterisks — always write a short temp script to the project's scratch dir and run it. Move it to tests/ or scripts/ if it becomes a standard op.
4. Use the TEST database for pentest/eval — never prod. The engine reads two URLs: the app-role URL (prod/serving) and the pentest URL (test). If the user says "test" or "eval", set BOTH the engine URL AND the app URL to the test DB, or the engine silently hits prod.
5. App role first. The engine must connect as a non-superuser app role with FORCE ROW LEVEL SECURITY — never a superuser. Superusers bypass RLS (lesson L1).

The recipe (READ before acting)

• State: docs/01_STATE.md — what exists, architecture map, what's proven
• Environment: docs/02_ENVIRONMENT.md — exact environment, DBs, .env layout
• Runbooks: docs/03_RUNBOOKS.md — copy-paste daily ops
• LESSONS (read before changing code): docs/04_LESSONS.md — the journal
• Enterprise gaps: docs/05_ENTERPRISE_GAPS.md — the roadmap
• Cheatsheet: docs/06_CHEATSHEET.md — one-page commands

If a recipe file is missing in this repo, the skills themselves below ARE the recipe — the structure (ground rules → runbooks → lessons → gaps) is the pattern.

Standard command flows

"Run the pentest" / "is it ready" / proof

python pa/pentest.py

• Expect PASS: 31 WARN: 0 FAIL: 0 ERROR: 0, exit 0.
• Report overwrites reports/PENTEST_REPORT.md.
• Any FAIL → read the lesson list below, don't guess.

RAGAS eval

python pa/ragas_eval.py --sample 10

• Expect Faithfulness ≈0.92, Context Precision/Recall 1.00, Hit@1 1.00.

Feed documents (ingest)

python ingest/raw_extractor.py --batch          # binaries → text
python ingest/pgvector_feed.py --domain <d> --source <dir>   # text → vectors

• De-dupe is by sha256 ledger / registry — re-running is safe and skips.

Provision a tenant

python tenant.py --name "Demo Clinic" --vertical klinik --plan "Ops"

• Creates tenants/<domain>/config.json + demo corpus + smoke questions
  • registry row. Default = TEST DB (safe). --prod for real.

Internal retrieval eval

python eval_set.py

• Expect high pass-rate; client domains 100%.

Triage order for "something's wrong"

1. Is the DB up? (docker compose ps)
2. Is the DB reachable AS the APP ROLE? (probe with app URL, not admin)
3. Run the pentest — which test fails? Read the FAIL line.
4. Wrong-but-passing retrieval → check eval cases + embedding cache (L6!).
5. Read the lessons — your bug is probably already there.

The 10 lessons (if anything looks wrong, it's one of these)

1. Superuser bypasses RLS — engine must run as a non-superuser app role + FORCE RLS.
2. Cached DB connection bleeds tenant GUC — re-pin the tenant on every call.
3. Documents can inject prompts — synthesis context is DATA, not instructions.
4. Empty context = hallucination — refuse before calling the LLM.
5. Provider failover must retry at RUNTIME (primary→fallback chain), not just at startup.
6. Embedding cache can CORRUPT retrieval (near-duplicate hit) — exact-key caching only; never text[:200].
7. RLS policy errors if GUC unset — pin tenant on every per-tenant table connection.
8. jsonb needs psycopg2.extras.Json, not a raw dict.
9. App role needs GRANT USAGE ON ALL SEQUENCES for SERIAL inserts + per-table CRUD.
10. Shell + inline scripts = broken; write a temp file.

What to do when a task isn't in the runbooks

• Read the enterprise-gaps roadmap — the unbuilt surround (SSO, user audit, web UI, monitoring, load tests). If the request touches a gap, say which one it maps to and build against the same RLS/answer-log patterns.
• When changing engine code, re-read the relevant lesson first. If you see "flaky" behavior, assume L6 first — it's the scariest failure mode (silent wrong answers), not the most visible.

Skill pattern: ground rules → runbooks → lessons → gaps. The value is not the code; it's the debugging that the next operator doesn't have to repeat.