# Sovereign RAG Ops

> Operations for a PROVEN, battle-tested multi-tenant RAG platform. Use when the task is "run the pentest", "is the product ready", "verify retrieval", "provision a tenant", "feed documents", "evaluate the RAG engine", or any day-to-day operation on a tenant-isolated retrieval engine. Bakes in the PROVEN recipe (runbooks + 10 hard-won lessons) so the agent executes the stack correctly without re-deriving anything.

- Skill: `amirulskulker/sovereign-rag-ops` (Agent Skill)
- Install (CLI): `npx skillmds@latest add amirulskulker/sovereign-rag-ops`
- Raw SKILL.md: https://api.skillmd.com/api/skills/amirulskulker/sovereign-rag-ops/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: amirulskulker (https://skillmd.com/u/amirulskulker)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/amirulskulker/sovereign-rag-ops

---


# Sovereign RAG — Operations Skill

You are operating a PROVEN, battle-tested RAG platform. Do not improvise —
follow the recipe. The recipe encodes a week of debugging so you don't
repeat it.

## Ground rules (non-negotiable)

1. **Use the project's pinned Python** (the one with psycopg2/pgvector/dotenv
   installed). The system default often lacks these. Check the project's
   `requirements.txt` / `.tool-versions` / README and use that interpreter.
2. **Never print secrets.** Passwords/tokens live in the project's untracked
   `.env` (or secrets vault). Mask anything credential-like (first 3 chars +
   length only). Never echo a full value into logs, chat, or files.
3. **Shell inline scripts break on quotes/asterisks** — always write a short
   temp script to the project's scratch dir and run it. Move it to
   `tests/` or `scripts/` if it becomes a standard op.
4. **Use the TEST database for pentest/eval — never prod.** The engine reads
   two URLs: the app-role URL (prod/serving) and the pentest URL (test). If
   the user says "test" or "eval", set BOTH the engine URL AND the app URL to
   the test DB, or the engine silently hits prod.
5. **App role first.** The engine must connect as a non-superuser app role
   with FORCE ROW LEVEL SECURITY — never a superuser. Superusers bypass RLS
   (lesson L1).

## The recipe (READ before acting)

- **State:** `docs/01_STATE.md` — what exists, architecture map, what's proven
- **Environment:** `docs/02_ENVIRONMENT.md` — exact environment, DBs, .env layout
- **Runbooks:** `docs/03_RUNBOOKS.md` — copy-paste daily ops
- **LESSONS (read before changing code):** `docs/04_LESSONS.md` — the journal
- **Enterprise gaps:** `docs/05_ENTERPRISE_GAPS.md` — the roadmap
- **Cheatsheet:** `docs/06_CHEATSHEET.md` — one-page commands

If a recipe file is missing in this repo, the skills themselves below ARE the
recipe — the structure (ground rules → runbooks → lessons → gaps) is the
pattern.

## Standard command flows

### "Run the pentest" / "is it ready" / proof
```bash
python pa/pentest.py
```
- Expect `PASS: 31  WARN: 0  FAIL: 0  ERROR: 0`, exit 0.
- Report overwrites `reports/PENTEST_REPORT.md`.
- Any FAIL → read the lesson list below, don't guess.

### RAGAS eval
```bash
python pa/ragas_eval.py --sample 10
```
- Expect Faithfulness ≈0.92, Context Precision/Recall 1.00, Hit@1 1.00.

### Feed documents (ingest)
```bash
python ingest/raw_extractor.py --batch          # binaries → text
python ingest/pgvector_feed.py --domain <d> --source <dir>   # text → vectors
```
- De-dupe is by sha256 ledger / registry — re-running is safe and skips.

### Provision a tenant
```bash
python tenant.py --name "Demo Clinic" --vertical klinik --plan "Ops"
```
- Creates `tenants/<domain>/config.json` + demo corpus + smoke questions
  + registry row. Default = TEST DB (safe). `--prod` for real.

### Internal retrieval eval
```bash
python eval_set.py
```
- Expect high pass-rate; client domains 100%.

### Triage order for "something's wrong"
1. Is the DB up? (`docker compose ps`)
2. Is the DB reachable AS the APP ROLE? (probe with app URL, not admin)
3. Run the pentest — which test fails? Read the FAIL line.
4. Wrong-but-passing retrieval → check eval cases + embedding cache (L6!).
5. Read the lessons — your bug is probably already there.

## The 10 lessons (if anything looks wrong, it's one of these)

1. Superuser bypasses RLS — engine must run as a non-superuser app role +
   FORCE RLS.
2. Cached DB connection bleeds tenant GUC — re-pin the tenant on every call.
3. Documents can inject prompts — synthesis context is DATA, not instructions.
4. Empty context = hallucination — refuse before calling the LLM.
5. Provider failover must retry at RUNTIME (primary→fallback chain), not
   just at startup.
6. Embedding cache can CORRUPT retrieval (near-duplicate hit) — exact-key
   caching only; never text[:200].
7. RLS policy errors if GUC unset — pin tenant on every per-tenant table
   connection.
8. jsonb needs `psycopg2.extras.Json`, not a raw dict.
9. App role needs `GRANT USAGE ON ALL SEQUENCES` for SERIAL inserts +
   per-table CRUD.
10. Shell + inline scripts = broken; write a temp file.

## What to do when a task isn't in the runbooks

- Read the enterprise-gaps roadmap — the unbuilt surround (SSO, user audit,
  web UI, monitoring, load tests). If the request touches a gap, say which
  one it maps to and build against the same RLS/answer-log patterns.
- When changing engine code, re-read the relevant lesson first. If you see
  "flaky" behavior, assume L6 first — it's the scariest failure mode (silent
  wrong answers), not the most visible.

---
*Skill pattern: ground rules → runbooks → lessons → gaps. The value is not
the code; it's the debugging that the next operator doesn't have to repeat.*