Tensor Grep Semantic Search Campaign

tensor-grep — Local Hybrid Semantic Search Campaign

A decision-gated runbook for building the APPROVED local hybrid semantic search layer: BM25 (lexical) + a CPU dense-embedding leg, fused with Reciprocal Rank Fusion (RRF), 100% local, no API key, no GPU. This is roadmap item #1 (grep -n "the #1 validated user ask" AGENTS.md; was :561-562 at v1.96.0, now :1738 at v1.101.27 — cite the grep, not the number) — the #1 validated user ask and the biggest competitive gap.

This skill is the campaign map. It tells you what already exists, what you are building, the exact commands + expected numbers at each gate, the wrong paths that are fenced off, and how promotion routes through change-control. You do not ship anything user-visible from this skill without beating the gate and doing a conscious flag-flip (see Phase 5).

STATUS UPDATE (2026-07-08, v1.49.3): SHIPPED. The dense leg + RRF fusion this skill specifies as the target architecture (Candidate 1, the Semble pattern) is now live: src/tensor_grep/core/retrieval_dense.py (model2vec + potion-code-16M, DenseUnavailableError/BackendExecutionError fail-closed contract exactly as §6 specifies) and src/tensor_grep/core/retrieval_fusion.py (reciprocal_rank_fusion(rankings, k=DEFAULT_K=60) — matches §3/§5 exactly), wired as tg search --semantic (grep -n '"--semantic"' src/tensor_grep/cli/main.py; was main.py:7069, now main.py:7331 typer option, default False; bootstrap front doors at bootstrap.py:70 [_TG_ONLY_SEARCH_FLAGS, keeps it off the rg-passthrough] and bootstrap.py:478 [_can_delegate_to_native_tg_search's unsupported_flags, keeps it off native-Rust delegation]) gated on the optional semantic extra (pyproject.toml:627, model2vec>=0.5+numpy>=1.26; all four citations re-verified 2026-07-24 against v1.96.0). Sections 1-2 below still describe the PRE-BUILD state and are now WRONG on the "does not exist yet" claims — read them as historical design intent, not current fact. No tg index command was added (the persisted-index building blocks in semantic_index.py remain unwired, per the original §1 note). RESOLVED (2026-07-24, re-verified against v1.96.0): --semantic has NOT graduated past default-OFF — SearchConfig.semantic_rank: bool = False (config.py:188) and AGENTS.md's own Roadmap Sequencing section still describes the shipped win as "default-OFF" (grep -n "the #1 validated user ask" AGENTS.md; was :561-562, now :1738). Still not re-verified: whether Phase 4's promotion gate (RRF-hybrid beats BM25-only on a real corpus + editor-plane latency) was actually measured before the ORIGINAL ship — that is a historical-PR question this pass did not chase down; if you need it, check the --semantic flag's introducing PR for attached benchmark evidence. If you are extending this further (chunking, a tg index command, a default-flip), Phases 4-5 below are still the right runbook.

Two further opt-in refinements SHIPPED since (both default-OFF, additive): TG_CHUNKER=structural (PR #443, 9015238, shipped v1.47.0) -- cAST AST-shaped chunking (retrieval_chunker.py, CHUNKER_MODE_ENV_VAR) beside the fixed-window chunker, fail-open, chunk-shape-identical contract, index-version-bumped (semantic_index.py v2 folds the active chunker mode into its cache key). This retires the old "cAST is a candidate deepener, not a requirement for v1" framing in S3 Candidate 1 below -- it shipped as an explicit opt-in, not a requirement, but it is no longer merely a future candidate. TG_RRF_CHANNELS=1 (PR #442, a402f81, shipped v1.46.0) -- channelized RRF (reranker.py, _RRF_CHANNELS_ENV): weighted per-channel fusion including a 1.5x path/filename channel, additive weights param, default-off and byte-identical when unset.

(Superseded note, kept for history: as of 2026-07-05/v1.40.2 this was still unbuilt and tensor-grep-large-repo-scale-campaign was the live campaign instead — that is no longer the case for the dense/RRF leg specifically; re-check which campaign is "live" at the time you read this.)

STATUS UPDATE 2 (2026-07-16, v1.77.0-v1.78.1, campaign #189): the architecture GRADUATED into a standalone whole-repo command, tg find. Where --semantic re-ranks an EXISTING regex match set, tg find walks and ranks the WHOLE repo (no pattern pre-filter), reusing the same retrieval_dense.py/retrieval_fusion.py core. It shipped its own golden harness, benchmarks/eval_late_rerank_quality.py (a 40-query NL vocab-mismatch golden set + literal/identifier golden slices, superseding this skill's toy eval_bm25_quality.py as the Phase-4-style discriminating gate for tg find specifically) — gate-run result: rrf beats bm25 by +0.195 ndcg@10 / +0.30 recall@10, bidirectional-oracle-validated (internal; public numbers stay CEO-gated #72). Two related dispositions are: (1) MaxSim late rerank (TG_LATE_RERANK) is RETIRED (task F10): the post-role-aware-encoder measurement was decisively negative; retrieval_late.py records the retirement. Do not re-run the same encoder expecting a different verdict. (2) TG_FIND_DENSE_WEIGHT (see tensor-grep-config-and-flags) now adaptively applies 5.0 to multi-word queries when unset, empty, malformed, or non-finite; single-token queries remain 1.0, and explicit TG_FIND_DENSE_WEIGHT=1.0 opts out. Receipt (real-corpus-dogfood-beats-fixture-green): the query classifier that scopes TG_FIND_DENSE_WEIGHT to multi-word queries was originally a split_terms() morpheme-count floor (> 2 morphemes = NL); it passed its synthetic literal-golden fixture but a real-repo dogfood on tensor-grep's own src/ caught it mis-boosting 5 of 6 literal identifier queries (_confine_mcp_path, getUserName, reciprocal_rank_fusion all split into 3+ morphemes) — fixed by switching to a whitespace word-count gate (len(query.split()) <= 1 stays literal), #191/#630. See tensor-grep-run-and-operate §1/§7/§11c for the CLI/MCP command surface and exit contract, and the dedicated operator skill tensor-grep-find-and-route for the day-to-day tg find/tg route-test CUJ (this skill stays the BUILD/campaign history; that one is the how-to-run doc).

STATUS UPDATE 3 (2026-07-21, research campaign #251) — cAST structural chunking REJECTED as default; do not re-propose it. TG_CHUNKER=structural (shipped v1.47.0, mentioned above as an opt-in refinement) was evaluated as a candidate for the DEFAULT chunker on a real-corpus retrieval eval: the retrieval-quality delta was a net WASH, while cAST chunking ran 24.4x SLOWER and produced ~38% LARGER chunks than the shipped line-window chunk_file. The opt-in code remains shipped for experimentation; it is not promoted, and this is now a documented retirement in docs/PAPER.md §3.10 — do not re-run this experiment expecting a different verdict without new evidence. Use the right harness when re-measuring anything chunker-sensitive: benchmarks/eval_late_rerank_quality.py (live — imports and calls chunk_file/chunk_file_structural) is the correct instrument; benchmarks/run_repo_retrieval_benchmarks.py is a static-fixture REPLAY that never calls chunk_file at all — it cannot detect a chunker regression or improvement and was mistakenly cited as the Phase-4 discriminating gate earlier in this skill (§0/Phase 4 below now point at the correct script).

One-line caution (do not re-propose): dense-embedding compression (int8/binary/PCA) was evaluated and DEFERRED — memory-only win (3.79x smaller on disk), but ~2x SLOWER in numpy (no int8 SIMD on the CPU-only path this campaign targets); a real speed win needs a native kernel (banked as a moat-investment option, not a quick follow-up here).

Sibling hardening, not a semantic-search change: #699/A7 hardened the SIBLING flat lexical scorer (_score_symbol in repo_map.py — exact word-boundary bonus + test-file demotion), which is a DIFFERENT scorer from this campaign's BM25/dense/RRF stack (see code-search-and-retrieval-reference §3) — do not conflate the two when reading a "ranking fixed" claim.

STATUS UPDATE 4 (2026-08-12 retention pass; verified against v1.110.14, base 568065a): the TG_FIND_DENSE_WEIGHT claim in STATUS UPDATE 2 item (2) above — "default-OFF (1.0 = byte-identical no-op)" with the flip "a separate, still-open CEO checkpoint" — is FALSE at this SHA. The flip SHIPPED (#191/#634, commit c1d4ba4); the CEO checkpoint is closed by it. Verified resolution rule (grep -n "_FIND_DENSE_WEIGHT_ADAPTIVE_DEFAULT" src/tensor_grep/cli/main.py — = 5.0 at main.py:4614; reader _find_dense_weight at main.py:4627-4684): unset / empty / malformed / non-finite env → the adaptive _FIND_DENSE_WEIGHT_ADAPTIVE_DEFAULT (5.0, the ledger-swept 1:5 bm25:dense ratio) for genuinely multi-word NL queries; a single whitespace-free token stays pinned at _FIND_DENSE_WEIGHT_DEFAULT (1.0) regardless of the env var's state; explicit TG_FIND_DENSE_WEIGHT=1.0 is the opt-out back to the old equal-weight fusion (any other finite value, e.g. =3.0, is honored verbatim). Evidence (grep -n "dense-weight flip" CHANGELOG.md): NL ndcg@10 0.3047->0.4466, zero per-category regression. Version note (verified, do not propagate the mislabel): the v1.93.4-era CHANGELOG #712 entry calls this "the v1.93.2 dense-weight flip", but git tag --contains c1d4ba4 puts the flip commit's FIRST release at v1.79.0 (its own CHANGELOG entry sits under ## v1.79.0 (2026-07-16)) — the v1.93.2 label in that later entry is a mislabel. The old item-(2) text above is kept as dated history: it was accurate for the #628/#630 flip-prep era and is closed by the flip.

SUPERSEDED (same 2026-08-12 pass): STATUS UPDATE 2 item (1)'s MaxSim / TG_LATE_RERANK "re-run the tg find gate" instruction is RETIRED — do NOT re-run. The module docstring is the authority: src/tensor_grep/core/retrieval_late.py:4-16 (note: core/, not cli/) records "RETIRED 2026-08-05 (task F10)". The post-role-aware-encoder-fix re-measurement HAPPENED and was decisively negative: ndcg@10 0.068 vs plain RRF 0.305 (docs/PAPER.md:469), root cause model capacity (the 17M-param int8 LateOn-Code-edge model's raw MaxSim ranking is statistically indistinguishable from random on in-repo code) — NOT the encoder wiring, so "re-flipping the same encoder will not change the verdict." The module is kept in place, not deleted (the core/reranker.py wiring is too load-bearing to touch for a demand-gated feature), reachable only behind the undocumented TG_LATE_RERANK=1 env var with no tg-command install path. Reopen only on BOTH a real tg-command install path AND a different encoder clearing the design doc's T8 golden-set thresholds — never off a partial win on either alone.

0. When to use this skill — and when to use a sibling instead

Use this skill when the task is the hybrid-retrieval build itself: adding a dense leg, RRF fusion, a persisted hybrid index, a tg index command, or measurably improving tg search --rank.

This skill never routes around change-control. Promotion is a change-control decision (Phase 5); this skill only produces the evidence that decision needs.

1. What already exists (the BM25 lexical leg is SHIPPED)

2026-07-08 correction: this section (and §2 below) was written when only the BM25 leg existed. The dense leg (retrieval_dense.py) and RRF fusion (retrieval_fusion.py) have SHIPPED since — see the STATUS box above. Treat every "does not exist yet" statement below as describing the pre-2026-07-0x state, not current fact; re-verify with the grep in the STATUS box before relying on it.

Read these before writing a line. Every path below is verified against the repo as of v1.17.25.

How --rank is wired (verify before changing):

• Flag: --rank (alias --bm25), default OFF. SearchConfig.rank_bm25 = False (config.py:183, re-verified 2026-07-24 against v1.96.0). The dense leg's own flag sits right below it: SearchConfig.semantic_rank = False (config.py:188).
• It is a TG-only search flag: bootstrap.py::_TG_ONLY_SEARCH_FLAGS (--rank line 68, --bm25 line 69, --semantic line 70 — re-verified 2026-07-24) — the bootstrap front door intercepts it and does NOT forward it to ripgrep. This is one of the two flag front doors; see tensor-grep-config-and-flags.
• Setting --rank leaves the ripgrep passthrough fast-path: the _can_passthrough_rg() condition includes and not config.rank_bm25 and and not config.semantic_rank (grep -n "not config.rank_bm25\|not config.semantic_rank" src/tensor_grep/cli/main.py; was main.py:5232-5250 at v1.96.0, now main.py:5438-5463 at v1.101.27 — this seam has already drifted twice inside two weeks, cite the grep, not the number), so the request runs the tg engine and results are re-ordered right after match aggregation — the elif config.rank_bm25 and all_results.matches: guard through the rerank_by_bm25(...) call (grep -n "elif config.rank_bm25 and all_results.matches\|rerank_by_bm25" src/tensor_grep/cli/main.py; was main.py:7965-8069; the follow-up :8411-8414 pin then landed INSIDE a --deadline option block deleted by the 2026-08-23 de-duplication, so it has no successor either -- the THIRD drift of this one anchor. Use the grep above and stop pinning it).
• User docs: grep -n -- "--rank" README.md (feature bullet :39, example :172-173 as of 2026-08-14; were :38 and :147-148 at the v1.96.0 pass).

Bottom line: the lexical leg (BM25) and the persisted-index building blocks already exist and ship default-OFF. The campaign adds the **dense leg + RRF fusion

• (optionally) a wired persisted hybrid index.**

2. What you are building (the approved architecture)

                 chunk_file()  ──►  chunks  ──►  ┌─ BM25Index.query() ──► ranking A
   query ──►  split_terms()                      │
                 (same tokenizer)                 └─ dense encode+cosine ──► ranking B
                                                          │
                                        RRF fuse(A, B, k=60)  ──►  final ranking

• BM25 leg — exists (retrieval_bm25.py).
• Dense leg — DOES NOT EXIST YET. A CPU static-embedding model produces a vector per chunk and per query; rank chunks by cosine similarity. Static means a per-token vector lookup (no transformer forward pass at query time) → fast on CPU, no GPU, no API key, no network at query time.
• RRF fusion — DOES NOT EXIST YET. Combine the two rankings without score normalization: score(d) = Σ_r 1 / (k + rank_r(d)) over the rankers r ∈ {bm25, dense}, with k = 60 (the value the reference architecture uses). A document absent from a ranker's list contributes 0 for that ranker. RRF is rank-based, so it is robust to the fact that BM25 scores and cosine scores are on incomparable scales.

SUPERSEDED (was true through v1.40.2, 2026-07-05; false as of v1.49.3, 2026-07-08): no dense/embedding/RRF/Model2Vec/potion code exists in src/ today — this leg has since shipped as retrieval_dense.py + retrieval_fusion.py; re-run grep -rin "model2vec|potion|reciprocal_rank_fusion|StaticModel" src/ yourself and expect real hits, not just comments.

The moat framing (do not lose it): this is not "faster grep." ripgrep is the raw-text parity baseline. The value is agent-native retrieval quality on vocabulary-mismatch queries (find authenticate when the user typed "verify login"). Keep the positioning honest per tensor-grep-release-and-positioning.

3. Solution menu (RANKED) with derivation obligations

Pick in this order. Each candidate carries a derivation obligation — a claim you MUST verify (not assume) before you build on it. "Derive" = confirm against a primary source (the model card, the license file, a local import test), then record the finding. Route the research through tensor-grep-research-frontier + tensor-grep-research-methodology; never trust a self-report (change-control gate B).

Candidate 1 (preferred): the Semble pattern

Tree-sitter chunking + potion-code-16M Model2Vec static embeddings + BM25 + RRF (k=60). CPU-only, MIT. This is the reference architecture named in AGENTS.md (grep -n "MinishLab .Semble." AGENTS.md; was :565 at v1.96.0, now :1741 at v1.101.27; the "## Roadmap Sequencing" heading it sits under was :525, now :1701 — AGENTS.md keeps growing new sections above these, cite the grep not the number).

Derivation obligations before you depend on it:

1. License — confirm potion-code-16M (and the model2vec runtime) are MIT/Apache-compatible with tensor-grep's Apache-2.0 and add the required NOTICE entries. Ideas are free; imported code/weights need their notices.
2. Truly offline — confirm the model loads from a bundled/cached file with no network call and no API key at query time. If it phones home or needs a token, it is DISQUALIFIED (see §4 fenced paths).
3. CPU + footprint — confirm it runs with no GPU, and record the on-disk model size and the added dependency weight. The model must be an OPTIONAL extra, not a hard install dependency (every-install must still work with BM25-only).
4. Chunking choice -- RESOLVED, both ship. Semble uses tree-sitter chunks; tensor-grep already had line-window chunk_file as the default, and now also ships TG_CHUNKER=structural (PR #443, v1.47.0) -- opt-in cAST AST-shaped chunking (docs/PAPER.md/arXiv:2506.15655 is the reference paper), fail-open and chunk-shape-identical to the fixed-window path when unset. This was a candidate deepener as of the original design; it is now a shipped, default-off refinement -- do not describe it as unbuilt.

Candidate 2: ripvec (pure-Rust)

A pure-Rust vector path. Derivation obligations: confirm license, maturity, and whether it fits the existing PyO3 bridge without reintroducing the FFI overhead that was already measured too high and reverted (tensor-grep-failure-archaeology: FFI is not the dir-scan speed path). Only choose this if Candidate 1 fails a gate AND you have measured that the Rust path clears the same promotion bar.

Candidate 3: BM25-only (the honest null result)

Ship nothing new. This is a legitimate, non-embarrassing outcome if the dense leg does not beat the BM25 baseline on both retrieval quality and editor-plane latency. "No speed/quality claim without measured numbers vs the baseline" (change-control gate C) cuts both ways: if the numbers aren't there, the correct move is to keep the shipped --rank baseline and record the negative result. grep -n -- "demonstrably beats" README.md -- :237 as of 2026-08-14 (was :212 at the v1.96.0 pass) states the rule explicitly: extend lexical (BM25) re-ranking with AST-shaped chunking or semantic re-ranking only when it demonstrably beats the shipped tg search --rank baseline on both retrieval quality and editor-plane benchmarks.

4. Fenced-off wrong paths (do NOT do these)

5. The phased runbook (decision-gated)

Run phases in order. Each gate says the expected number and where to branch if you see something else. All commands are copy-pasteable; PowerShell is the primary shell on the dev box, but uv run is cross-platform.

uv run gotcha: a bare uv run ... can re-sync and drop the [dev] tree-sitter/extras tree. For benchmark/import work use uv run --no-sync ... so the installed dev tree (and any editable install of the dense model) is not wiped. See tensor-grep-build-and-env.

Phase 0 — Establish the baseline (NEVER skip)

You cannot claim an improvement without the number you improved on.

uv run --no-sync python benchmarks/eval_bm25_quality.py --top-k 3

Expected (verified 2026-07-02):

BM25 baseline (top_k=3, n=10 queries):
  recall@k   = 1.000
  precision  = 0.333
  mrr@k      = 1.000
  ndcg@k     = 1.000
v2 gate (recall@k >= 0.6): PASS

• benchmarks/eval_bm25_quality.py defines V2_GATE_RECALL = 0.60 and states in its own docstring: "the v2 dense+RRF leg must beat this before it ships user-visible."
• GATE 0a: recall@k must be 1.000 here. If it is lower, the BM25 leg itself regressed → STOP, do not build dense on a broken base → tensor-grep-debugging-playbook.
• GATE 0b — read this or you'll waste weeks: this toy corpus is keyword-discriminating, so BM25 already saturates it at recall 1.0. Passing the 0.60 gate proves nothing about whether dense helps — it is a floor / sanity gate, not the discriminating gate. The real justification for a dense leg is vocabulary-mismatch queries (synonyms/paraphrase) where lexical BM25 misses. Your promotion evidence MUST come from a harder, realistic corpus (Phase 4), not this file.

Phase 1 — Choose + derive the solution

Work the §3 menu top-down. Complete every derivation obligation for your chosen candidate and write the findings down (license, offline-proof, CPU-proof, footprint, optional-extra plan). Research via tensor-grep-research-frontier + tensor-grep-research-methodology. Before writing code, verify the plan's seam claims against the real files with file:line citations (verify-plan-against-code) — an AI-drafted plan that says "add it in pipeline.py" is a hypothesis until you confirm --rank is actually wired in main.py.

• GATE 1: if the preferred candidate fails a derivation obligation (needs a key, needs a GPU, incompatible license, cannot be an optional extra), do not "work around it" — drop to the next candidate. If all dense candidates fail, Candidate 3 (BM25-only) is the correct answer; document the negative result and stop.

Phase 2 — Build the dense leg behind a default-OFF experimental flag

• Add a new module (mirror the existing seam names, e.g. core/retrieval_dense.py) — do not bolt onto retrieval_bm25.py.
• Reuse split_terms() (or an explicitly justified tokenizer) so the two legs stay comparable.
• Wire it behind a separate default-OFF flag (the config.py:182 note already anticipates "a separate gated flag" for the dense leg). Do NOT change the meaning of --rank/--bm25 yet.
• Honor the Backend Fail-Closed Contract (§6).
• TDD: write the contract test first (tests/unit/test_retrieval_dense.py), then the smallest implementation. See existing tests tests/unit/test_retrieval_bm25.py, tests/unit/test_reranker.py, tests/unit/test_semantic_index.py for the pattern.
• GATE 2: the dense leg imports and runs with the model absent (degrades to BM25-only, visibly) AND with it present (produces a ranking). Prove the FFI/model path against the REAL runtime, not a mock — mock-green while the real bridge is dead is a documented trap (tensor-grep-failure-archaeology).

Phase 3 — Build RRF fusion

• Implement reciprocal_rank_fusion(rankings, k=60) as a pure function taking each leg's ordered list of chunk indices and returning the fused order. Keep k configurable (default 60) and deterministic ties.
• Fuse ranks, not raw scores (BM25 score vs cosine are incomparable scales — this is the whole reason RRF is chosen).
• GATE 3: unit-test that fusing two identical rankings is a no-op, and that a document top-ranked by either leg surfaces near the top of the fused list.

Phase 4 — Measure (the real gate)

Two measurements, both required (grep -n -- "demonstrably beats" README.md -- :237 as of 2026-08-14, was :212):

1. Retrieval quality on a realistic corpus (not the toy). Use benchmarks/eval_late_rerank_quality.py — the LIVE, chunker/ranking-sensitive harness (it actually imports and calls chunk_file/rank_chunks, and computes RetrievalMetrics: recall/precision/ mrr/ndcg on a real repo + the 40-query NL golden set). Do not use benchmarks/run_repo_retrieval_benchmarks.py for this — it is a static-fixture REPLAY that never calls chunk_file and cannot detect a chunker or dense/RRF-weighting change at all (this was the cAST-chunking evaluation's own harness-selection mistake before STATUS UPDATE 3 above corrected it). Produce three rows: BM25-only, dense-only, RRF-hybrid, on the SAME corpus + queries.

   uv run --no-sync python benchmarks/eval_late_rerank_quality.py --output artifacts/bench_find_quality.json
   

   (Read its args first; see tensor-grep-benchmark-and-proof-toolkit §7 for the corpus-hardness and paired win/loss/tie reporting rigor this gate specifically needs.)

2. Editor-plane latency — the ranking overlay must not blow the interactive budget:

   uv run --no-sync python benchmarks/run_editor_plane_benchmarks.py
   

Route interpretation through tensor-grep-benchmark-and-proof-toolkit (noise-floor rule for sub-10ms rows, fair-baseline rule, launcher attribution). Also record the token-economy delta if the surface feeds the agent capsule (AGENTS.md names token economy as a focus).

Promotion gate (all must hold, measured vs the SAME baseline run):

• GATE 4: if RRF-hybrid does not beat BM25-only on retrieval quality AND hold the line on latency, do not ship it. Reject the regression even if the code is clean (change-control gate C). The honest outcomes are: (a) improve the dense leg/chunking and re-measure, or (b) record the negative result and keep BM25-only.

Phase 5 — Promote through change-control (never here)

This skill produces evidence; tensor-grep-change-control owns the flip. The graduation path is fixed:

1. Ship experimental, default-OFF (already true after Phase 2).
2. Attach the Phase 4 evidence (three-row quality table + editor-plane numbers + token economy) to the PR.
3. Dry-run on real data (dogfood the REAL binary via scripts/dogfood/; CliRunner bypasses the bootstrap front door and will not exercise routing).
4. Conscious flag-flip — a deliberate, reviewed default change, never auto-merged, never admin-merged. Autonomy is draft-PR-only.
5. Update docs (README.md, AGENTS.md, the usage skill) via tensor-grep-docs-and-writing; observe one-merge-per-tick + the push-race rules (tensor-grep-release-and-positioning).

If you add a tg index command (the natural home for a persisted hybrid index), remember it needs the 4 command-registration sites (miss one → silent misroute) and a new flag needs the 2 flag front doors — see tensor-grep-config-and-flags and tensor-grep-architecture-contract.

6. Backend Fail-Closed Contract for the dense leg

The dense leg is a compute path; it is bound by backends/base.py (BackendExecutionError) and the AGENTS.md §"Backend Fail-Closed Contract" contract (grep -n "Backend Fail-Closed Contract" AGENTS.md; was :496 at v1.96.0, now :1672 at v1.101.27). The recurring anti-pattern to avoid: a bare except Exception: that silently returns empty or swaps engines.

• Model missing / not installed → this is a legitimate degraded fallback to BM25-only, but it MUST be VISIBLE: set a fallback_reason on the SearchResult so JSON/CLI consumers can tell degraded output from full hybrid output. Never label BM25-only output as "semantic."
• Model load/encode raises at runtime (corrupt cache, OOM, version skew) → raise BackendExecutionError; do not return a clean empty result that reads as "no matches." A real failure reported as a no-match is the exact bug this contract exists to prevent.
• Contract flag the fallback cannot honor → fail closed (raise), do not swap. (For a ranking overlay a graceful visible degrade to BM25 is the norm; only fail-closed if a caller explicitly demanded semantic-only and you cannot deliver it.)
• Validate the model's output shape (vector dimensionality, chunk count) before you index, so a mismatch degrades gracefully instead of raising an IndexError that a broad except then swallows.

See tensor-grep-architecture-contract for the full contract and the planned SafeBackendMixin conformance gate.

7. Common failure modes → branch

8. Pre-flight checklist (before you open the PR)

• Phase 0 baseline recorded (eval_bm25_quality.py) — with the note that it's a floor, not the discriminating gate.
• Chosen candidate's derivation obligations all verified + written down (license, offline, CPU, optional-extra).
• Dense leg is a NEW module, behind a NEW default-OFF flag; --rank/--bm25 semantics unchanged.
• Fail-closed contract honored: missing model → visible fallback_reason; runtime error → BackendExecutionError.
• RRF is rank-based, k=60 default, deterministic; unit-tested (identity no-op + top-surfacing).
• Phase 4 evidence: three-row quality table (BM25 / dense / RRF) on a REAL corpus + editor-plane latency + token economy — RRF beats BM25-only on quality without a latency regression, OR the negative result is documented and you stop.
• --json / --ndjson / --format rg unchanged when ranking is off.
• Local validation green: uv run ruff check . · uv run ruff format --check --preview . · uv run mypy src/tensor_grep · uv run pytest -q (CI runs ruff format --check --preview — you MUST pass --preview).
• Real-binary dogfood, not just CliRunner (scripts/dogfood/).
• Promotion routed through tensor-grep-change-control; draft PR only; conscious flag-flip, never auto-merge.

Provenance and maintenance

Everything below is verifiable from the repo. Re-run these when a claim may have drifted; date-stamp any change.

• Version / date: facts originally verified v1.17.25 (2026-07-02); re-verified UNCHANGED against released v1.40.2 (origin/main 8829441) on 2026-07-05; spot-checked again 2026-07-08 against v1.49.3 and found the dense/RRF leg now SHIPPED (see STATUS UPDATE near the top); spot-checked again 2026-07-16 against v1.78.1 and found the architecture graduated into tg find (see STATUS UPDATE 2 near the top); spot-checked again 2026-07-22 against v1.93.2 and recorded the cAST-chunking rejection + dense-int8 deferral + install-dense row (see STATUS UPDATE 3 near the top — this was targeted at the research-campaign #251 retirements and the harness-selection correction, not a full re-walk of Phases 0-8 below). **Spot-checked again 2026-07-24 against v1.96.0 (origin/main 29cf59f): deleted the Provenance bullet below that directly contradicted the "Dense leg + RRF now shipped" bullet (its own grep no longer produces the "expect no hits" result it claimed — 5 real hits in main.py/config.py/reranker.py/retrieval_dense.py/retrieval_fusion.py); resolved the open "has --semantic graduated past default-OFF" question (NO — see the STATUS UPDATE box near the top); re-verified and fixed drifted file:line citations throughout §1/§3/§4/§6/ Provenance (main.py/bootstrap.py/pyproject.toml/README.md/AGENTS.md line numbers had each drifted anywhere from ~10 to ~1500 lines since the v1.17.25-era pins — main.py's _can_passthrough_rg moved the most, 3883→5249); fixed INDEX_VERSION=1 → the now-current INDEX_VERSION=2; and corrected a stale STATUS UPDATE 2 claim (the retrieval_late.py doc-role-encoder harness gap it cited as blocking TG_LATE_RERANK was fixed by #189 Item 1 since that note was written; the later F10 measurement retired MaxSim). This was a targeted re-verification of THIS skill's own claims, not a full re-walk of every sibling skill or every historical receipt (e.g. the 2026-07-16 +0.195 ndcg@10 gate-run number in STATUS UPDATE 2 is a dated point-in-time receipt, left as-is). Skill-library drift audit, 2026-08-01, against v1.101.27: every main.py/AGENTS.md file:line citation re-grepped again and eve

…(truncated)