Knowledge Workflow

Knowledge Workflow

Build the vault, then repair retrieval in measured rounds until it stops improving.

Standing Mandates

• The loop is the deliverable, not the edit. Every round is: measure → diagnose → change one thing → measure against the saved run. A round with two changes cannot be attributed or reverted, so it does not count as a round.
• Split before the first repair. Record --split holdout once, up front. A holdout measured after repairs proves nothing, and tuning against it destroys the only unbiased number in the loop.
• Fix the largest lever first, not the easiest edit. The order below is measured, not intuited. Vocabulary work feels productive and has repeatedly moved nothing while the bottleneck sat in the provider or the ranker.
• A stop condition is part of the run. Say which one applies before the round starts. A loop with no declared exit becomes editing that looks like progress.
• Never cite the round count as progress. Report the holdout number, or say it did not move.

Workflow Overview

[A] Build once
  catalog → graph → RAG → index
        ↓
[B] Split + baseline   ← record holdout, save the run
        ↓
   ┌──> [C] Diagnose  (repair_targets, kind breakdown)
   │         ↓
   │    [D] One change (lever order below)
   │         ↓
   │    [E] Re-measure vs baseline → keep or revert
   │         ↓
   └─── stop condition met? ──no──┘
                 │ yes
                 ↓
        [F] Report + hand off to knowledge-query

Phase A — Build (once)

Competency questions in _knowledge/questions.jsonl are a precondition for Phase B, not an output of it. Without them there is nothing to measure and the loop cannot start.

Phase B — Split and baseline (once)

node --no-warnings "${CLAUDE_PLUGIN_ROOT}/scripts/sqlite-knowledge.mjs" eval \
  --root <vault> --k 10 --split holdout          # record this number, then leave it alone
node --no-warnings "${CLAUDE_PLUGIN_ROOT}/scripts/sqlite-knowledge.mjs" eval \
  --root <vault> --k 10 --split dev > /tmp/round-0.json

Record the reranker ceiling once too: eval --k 50 minus eval --k 10. Required notes missing at both depths can never be recovered by reordering.

Phase C — Diagnose

Read, in this order: repair_targets (each gap and how many questions the note blocks), the per-kind hit breakdown, then relation_participant_evicted_ids on any comparison question that lost a sibling note.

Re-indexing between rounds is cheap: an unchanged document reuses its stored embedding, so a round that edits two notes re-embeds two documents. embeddings_computed far above the number of edited notes means the cache was rejected — usually because the provider or model changed, which makes the round a provider change, not a corpus one.

Phase D — One change

Apply exactly one. Ground every added term in the source, never in the question set — copying a question's words into a note is test-set leakage that scores well and retrieves nothing new.

Phase E — Re-measure

node --no-warnings "${CLAUDE_PLUGIN_ROOT}/scripts/sqlite-knowledge.mjs" eval \
  --root <vault> --k 10 --split dev --baseline /tmp/round-N.json

Corpus and vocabulary edits: revert on any regression, even when recall_at_k rose — competency sets are small enough that an aggregate gain routinely hides a question that stopped working. Engine, provider, and reranker changes are judged differently: they move every question at once, so weigh net movement and holdout together and name the regressions instead of reverting on their existence.

Stop conditions

1. Two consecutive rounds with no dev improvement — the remaining gaps are not the kind this loop fixes.
2. Every remaining repair_target is missing-note with no source material — an authoring problem; the honest answer is that the vault does not contain the answer.
3. The target recall is met — score the holdout once and report both numbers.
4. The holdout has not moved while dev keeps rising — stop and say so. That gap is overfitting, and more rounds widen it.

Output Template

## Retrieval round N

**Change:** <the one thing>       **Lever:** <provider|ranking|corpus|vocabulary>
**Dev:** hits A → B, recall@10 X → Y, MRR P → Q
**Verdict:** improved | regressed | unchanged
**Regressions:** <question ids, or none>
**Decision:** kept | reverted — <why>

**Holdout (recorded at round 0):** <number, unchanged since>
**Stop condition:** <which one, and whether it is met>

What Claude Does / What You Do

Related Skills

• Steps: knowledge:knowledge-base-builder, knowledge:knowledge-graph-builder, knowledge:rag-corpus-builder, knowledge:sqlite-index-builder
• After: knowledge:knowledge-query — answer from the index the loop stabilized
• Also: knowledge:render-graph-view, knowledge:ontology-builder
• Measured lever sizes and what is deliberately not being done: ROADMAP.md