# Code Search And Retrieval Reference > Use when you need the domain theory behind tensor-grep's search/retrieval behavior, not just the command syntax — ripgrep exit codes (including the exit-2-but-kept partial-results contract)/PCRE2/binary-NUL-detection/-uuu/-- sentinels, ast-grep + tree-sitter routing, the multi-language symbol-graph registry's fail-closed grammar-missing contract, BM25 vs the flat no-IDF capsule scorer, PageRank vs in-degree centrality, the trigram index, PyO3 + the GIL, MCP argv surface, LSP 3.17 framing, and Model2Vec/potion-code (SHIPPED as `tg search --semantic`). Load before explaining WHY tg behaves a certain way, reasoning about the protocol/algorithm THEORY underneath a backend/router change (exit-code semantics, scoring math, wire framing), or writing docs that touch these subsystems — for the invariants a backend/router change must not break, use `tensor-grep-architecture-contract` instead (or in addition). Not a how-to-run or how-to-debug guide — see the sibling table below for those. - Skill: `oimiragieo/code-search-and-retrieval-reference` (Agent Skill) - Install (CLI): `npx skillmds@latest add oimiragieo/code-search-and-retrieval-reference` - Raw SKILL.md: https://api.skillmd.com/api/skills/oimiragieo/code-search-and-retrieval-reference/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: oimiragieo (https://skillmd.com/u/oimiragieo) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/oimiragieo/code-search-and-retrieval-reference --- # Code Search & Retrieval Reference The domain-theory pack a mid-level engineer (or a model working cold) usually lacks, narrowed to **only the slice that governs tensor-grep's actual behavior**. Every claim below cites the tg file that uses it — read that file before relying on the claim in a review or a fix, because code drifts and this document does not update itself. Verified against the repo **as of 2026-07-08, v1.49.3**; §9's `tg find` addition and the new §10 (query-shape classification) verified **2026-07-16, v1.78.1**; §3's `_score_symbol`/`_symbol_rank_key` breakdown re-verified and corrected **2026-07-22, v1.93.2**. **Every `file:line` citation in the document re-grepped against `origin/main`, §2's AST-routing description corrected, and new §2a (the `lang_registry` symbol-graph tier) added, 2026-07-23, v1.95.0** — see the dated note at the end of "Provenance and maintenance" for what changed and why. **2026-08-01, v1.101.27: §2a's language-tier claim corrected (all 10 top-10 languages are registered, not 8; there is no unregistered-C/C++ gap) and every drifted `file:line` citation in the document converted to a `grep ` instruction carrying a `was -> now` receipt** — see the final dated note in "Provenance and maintenance" for the full list and for why a "verified as of DATE" tag on a line number does not stop it from rotting. ## When NOT to use this skill (use a sibling instead) | You need... | Use instead | |---|---| | Command syntax / which flag to pass | `.claude/skills/tensor-grep/SKILL.md` + `REFERENCE.md` | | The front-door/registration/fail-closed **contract** (the "must-hold" invariants) | `tensor-grep-architecture-contract` | | How to change code safely (gates, one-merge-per-tick, TDD) | `tensor-grep-change-control` | | A live bug you're actively debugging right now | `tensor-grep-debugging-playbook` | | "Has this already been tried and lost?" | `tensor-grep-failure-archaeology` | | Which benchmark script proves a speed/quality claim | `tensor-grep-benchmark-and-proof-toolkit` | | Env var / flag default values | `tensor-grep-config-and-flags` | | Build/toolchain setup | `tensor-grep-build-and-env` | | Building/extending the approved BM25+dense+RRF semantic-search roadmap item | `tensor-grep-semantic-search-campaign` | | Day-to-day `tg` invocation syntax (orient, search --rank, session, mcp, ...) | `tensor-grep-run-and-operate` | This skill explains **why** a subsystem behaves the way it does; it does not tell you how to run, fix, or ship it. --- ## 1. ripgrep internals — the cold-path baseline `rg` is tg's raw-text-search comparator and, for most default `tg search` invocations, the actual execution engine underneath (`RipgrepBackend`, `src/tensor_grep/backends/ripgrep_backend.py`; native Rust passthrough in `rust_core/src/rg_passthrough.rs`). Getting rg's edge semantics wrong means either a silent wrong-answer or a parity-test false green. **Exit codes are not binary.** ripgrep uses three, and tg's contract is to *match rg's exit code exactly*, including the non-obvious one — and exit `2` itself is not simply pass-or-raise, because of the partial-results contract (see `tensor-grep-architecture-contract` for the full `SearchResult.result_incomplete` story, added #341): | rg exit code | Meaning | tg's handling | |---|---|---| | `0` | at least one match found | pass through | | `1` | search ran cleanly, zero matches | **not** an error — a `SearchResult` with 0 matches, never raised as a failure | | `2`, matches parsed (soft per-file error, e.g. one unreadable path among many) | rg still emitted matches for the readable files | tg **keeps** those matches, sets `result_incomplete=True` + an `incomplete_reason`, and does **not** raise — `partial = result.returncode == 2 and total_matches > 0` (grep that exact expression in `ripgrep_backend.py` -- was `:123`, now `:124`, mirrored in `_search_files_with_matches`/`_search_counts`) | | `2`, nothing parsed, or any `> 2` | a genuine fatal failure (bad regex, unreadable path with no other matches, etc.) | `RipgrepBackend.search()` raises `BackendExecutionError` whenever `result.returncode > 1 and not partial` (grep `result.returncode > 1 and not partial` in `ripgrep_backend.py` -- was `:124-128`, now `:125-129`; the two sibling methods raise the same way -- was `:297` and `:413`, now `:308` and `:426`; the rg-missing guard -- was `:505`, now `:541` -- too). **RESOLVED #79/#10/#14 (commit `a7c9431`)** -- every `RipgrepBackend` fatal path used to raise a bare `RuntimeError`, deliberately not `BackendExecutionError`, so it would not get caught by `cli/main.py`'s `except BackendExecutionError:` per-file CPU-fallback retry; the fix flipped all of them to `BackendExecutionError` so that retry now catches rg failures the same way it does every other backend, per the Backend Fail-Closed Contract's normal convention. | **Do not describe exit-2 as unconditionally "raise `BackendExecutionError`"** — that was true before #341 (round-4 slice 3) and is no longer true; a partial exit-2 with real matches is now a **kept** result, not a failure. `tests/e2e/test_rg_parity_edges.py::test_rg_exit_code_edges_match` is parametrized exactly on the fatal-failure boundary (grep `ids=\["match", "no-match", "parse-error",` in the same file -- was line `149`, now `169`) and asserts `tg.returncode == rg.returncode` for every case (grep `def _assert_same_rg_behavior` -- was line `81`, now `96`) — a regex `(` (unbalanced paren, nothing parsed) must still make tg exit `2`, same as rg. Partial-parse-with-matches is covered separately by `tests/unit/test_rg_exit2_partial.py`. **PCRE2 is a different engine, not a flag.** rg's default matcher is the Rust `regex` crate — linear time, but **no lookaround, no backreferences**. `--pcre2`/`-P` switches to libpcre2, which supports both but requires the *pattern* to be valid UTF-8 (it transcodes). tg detects PCRE2 support with a real smoke test — build help output contains `--pcre2`/`PCRE2`, then actually run `rg -P "a(?=b)" -V` and check `returncode == 0` (grep `def supports_pcre2` in `ripgrep_backend.py` -- was `:27-51`, now `:54-77`). This matters because of the **fail-closed contract**: `--pcre2` routed through an engine that cannot honor PCRE2 semantics must raise, never silently execute as a plain-regex search that returns wrong (or merely different) matches (`src/tensor_grep/backends/base.py:7-14`, `BackendExecutionError`; grep `**Fail closed**` in `AGENTS.md` -- was line `444`, now `:2096`). A prior incident shipped exactly this bug — a broad `except Exception: pass` around the Rust passthrough silently ran `--pcre2` through the non-PCRE2 Python-regex engine — fixed in v1.17.17/18 (see `tensor-grep-change-control`, grep `## Part 4` in that skill's `SKILL.md` -- was lines `125-134`, now `:287-311` -- Backend fail-closed contract). **Binary detection is NUL-byte sniffing, and it changes exit codes.** rg's default binary heuristic scans early file bytes for a `\0`; on a hit, the file is treated as binary and searched under the binary-skip policy unless `-a`/`--text` is passed (`rg_contract.py` row `"text"`, `public_flags: ("-a", "--text")`). The parity fixture builds exactly this case — `binary_path.write_bytes(b"needle\0binary tail\n")` (grep that exact line in `test_rg_parity_edges.py` -- was `:41-43`, now `:44`) — and the `"binary-skip"` parametrization (grep `ids=\["match", "no-match", "parse-error",` -- was line `149`, now `169`) asserts tg's exit code matches rg's on a NUL-containing file, not just its stdout. **`-u`/`-uu`/`-uuu` are not blind passthrough here.** Upstream, each additional `-u` widens scope (`-u` = `--no-ignore`, `-uu` = `--no-ignore --hidden`, `-uuu` = `--no-ignore --hidden --binary`). tg's Python front door specifically *detects* any `-u*` flag (or `--unrestricted`, or an explicit no-ignore/hidden flag) as a request for unrestricted scanning and routes it through a broad-root safety guard (grep `def _search_args_request_unrestricted_generated_scan` in `bootstrap.py` -- was `:581-590`, now `:726-732`). This exists because of a real v1.13.1 incident: an unguarded broad-root unrestricted scan could recurse into `node_modules`/`.git`/multi-project workspace roots. If you're adding a new flag that widens scan scope, check whether it needs to join this guard's flag set — a missed case is a silent safety regression, not just a slow query. **`--` and `-e` matter for argv safety, not just POSIX correctness.** `--` ends option parsing so a user- or LLM-supplied pattern beginning with `-` cannot be reinterpreted as a flag (CWE-88 / the MCP-276 CVE class). tg's MCP tool handlers build subprocess argv with an explicit `--` sentinel before positionals for exactly this reason: `command.extend(["--", pattern, path])` (grep that exact call in `src/tensor_grep/cli/mcp_server.py` -- was `:1306`, now `:1375`, comment two lines above: "round-3 security: end options before the user-controlled positionals so a pattern beginning with `-` cannot be parsed by the native binary as a flag"). Note the narrower scope of this fix: it blocks *flag* injection via a missing `--`, not shell injection (list-argv subprocess calls already block shell injection). See `tensor-grep-change-control` before touching any subprocess argv builder. **BOM handling is a real, previously-broken seam.** UTF-8 BOM bytes at the start of a file/scenario JSON broke PowerShell-generated fixtures until scenario loading switched to `utf-8-sig` (`docs/PAPER.md:840`). AST rewrite's batch-apply path explicitly preserves BOM/CRLF through atomic writes (`docs/harness_api.md:700`, "batch apply reuses the same atomic-write, BOM/CRLF preservation, binary-skip, and stale-file protections as single rewrites"). **Open / candidate — verify before relying on these, no in-repo citation exists yet.** Two upstream ripgrep quirks worth probing against tg's passthrough the next time someone does a round-4-style edge sweep: (a) `--multiline --pcre2 --json` reportedly emits a single match with two submatches rather than two matches (upstream ripgrep issue tracker; exact issue number unverified against this repo — re-check before citing a number); (b) `rg -c` (count mode) reportedly omits files whose content contains a NUL byte from the count entirely, rather than counting them normally. Neither has a tensor-grep test fixture as of this writing — treat as **candidate**, not fact, until one exists. --- ## 2. ast-grep + tree-sitter — structural search Two backends implement structural (AST-aware, not text-regex) search and rewrite, and the router picks between them per query: | Backend | File | Availability gate | |---|---|---| | `AstBackend` (native) | `grep -n "^class AstBackend" src/tensor_grep/backends/ast_backend.py` -- `:142` as of 2026-08-14 (was `:133`) | `is_available()` checks **only** whether `tree_sitter` is importable (`grep -n "def is_available" src/tensor_grep/backends/ast_backend.py` -- `:608` as of 2026-08-14, was `:505-519`) — **no GPU/CUDA/`torch_geometric` gate anymore.** That gate was real once (see the "Corrected" note below) but was deleted in #542 (v1.65.0, 2026-07-12); the current docstring says outright: *"AstBackend.search() is pure tree-sitter query matching -- it never touches torch, CUDA, or any graph-learning library ... gating a fully-functional CPU backend behind an unrelated GPU dependency was itself the bug."* | | `AstGrepWrapperBackend` (sidecar) | `src/tensor_grep/backends/ast_wrapper_backend.py:85` | shells out to an installed `ast-grep`/`sg`/`sg.exe`/`ast-grep.exe` binary via `shutil.which` (lines 111-123) | `tree-sitter` parses source into a concrete syntax tree; a **metavariable** like `$FUNC` or the "capture the rest" form `$$$ARGS` is ast-grep/tg's pattern-matching primitive over that tree (e.g. `def $FUNC($$$ARGS):` matches any Python function definition, binding `$FUNC` and `$$$ARGS`). **Corrected — the routing default is the OPPOSITE of what this section previously said.** The real routing decision lives in `_select_ast_backend_for_pattern` in `src/tensor_grep/cli/ast_workflows.py` (grep `def _select_ast_backend_for_pattern` in `ast_workflows.py` -- the `main.py` copy this instruction used to point at (was `:6737`, then `:6915`) is now a THIN FORWARDING SHIM onto the ast_workflows implementation; `grep "Thin forwarding shim onto" src/tensor_grep/cli/main.py` finds it, and its docstring records the drift that forced the collapse -- the old hand-maintained duplicate silently dropped the `requires_ast_grep_wrapper` fail-closed guard. A dated hedge does not stop a line number from rotting, it just makes the rot look supervised, so no citation in this document carries a date anymore), and its own comment is unambiguous: *"Prefer the ast-grep wrapper whenever it is available: it is the stable, results-defining backend for BOTH pattern kinds. The native tree-sitter AstBackend uses a DIFFERENT DSL and returns DIFFERENT results, so it must not be silently preferred ... Native-as-CPU-default is task #141. Native is reached ONLY as the ast-grep-absent fallback for native patterns."* (grep "Prefer the ast-grep wrapper" in `ast_workflows.py` -- was `:6692-6697` in `main.py`, then `:6952-6957`; now `ast_workflows.py:1231-1232` (grep "Prefer the ast-grep wrapper")). Concretely: the wrapper availability check runs **first** (`if _check_backend_available("AstGrepWrapperBackend"): backend = _get_cached_backend("AstGrepWrapperBackend")` -- grep `_check_backend_available("AstGrepWrapperBackend")` in `ast_workflows.py`; the old `if ast_wrapper.is_available(): backend = ast_wrapper` shape this entry used to cite in `main.py`, was `:6698`, then `:6958`, is gone), unconditionally, regardless of GPU/CUDA or pattern shape; native `AstBackend` is reached only when the wrapper is unavailable AND the pattern qualifies as "native" (`base_config.ast_prefer_native` and `is_native_ast_language(base_config.lang)` — the native-capable language set is narrower than the wrapper's, `_NATIVE_AST_LANGUAGES = ("python", "javascript", "typescript", "tsx", "rust")`, `ast_backend.py:104`). This applies to AST search, `--rewrite` planning, `--apply`, `--diff`, and batch rewrite flows alike. **`docs/routing_policy.md`'s own "AST commands" section is itself stale here** (grep `## AST commands` in that doc -- was `:107-113`, now `:156`) — it still describes a `torch-geometric`/CUDA-style gate — trust the code cited above, not that doc's prose, until it is refreshed. **Practical corollary (updated):** on almost any real dev or CI box, `ast-grep`/`sg` being installed is now the thing that decides the backend, not GPU presence — with ast-grep installed (the common case, since it's how most people actually got `tg run` working), `tg run`/AST calls use the CLI sidecar regardless of CUDA. This *used* to be a CUDA story: an earlier "AST probe" bug in CI traced back to `AstBackend` being GPU-gated (see `tensor-grep-docs-and-writing` Part 6, and project memory `tensor-grep-readme-release-blocker-2026-06-25`, for that incident) — but the GPU gate itself is gone (#542 above), so don't cite CUDA absence as the reason native is skipped anymore; cite wrapper availability instead. Don't assume native AST speed numbers apply unless you've confirmed the *wrapper* is unavailable (or `--lang`/pattern shape forced native) on the box you're measuring. Measured (not marketing) ratios, `benchmarks/run_ast_benchmarks.py` / `run_ast_multilang_benchmarks.py` (`docs/benchmarks.md` "ast-grep vs tensor-grep AST mode"): single- query `tg 0.116s` vs `sg 0.151s` (`0.770x`); multi-language ratios Python `0.722x`, JavaScript `0.800x`, TypeScript `0.726x`, Rust `0.715x` — `tg` ahead of `sg` on all four **when the native path is actually reachable**. Given the routing correction above, treat these specific numbers as unverified-current: `run_ast_benchmarks.py` invokes the real `tg` binary end-to-end (`build_tg_ast_benchmark_cmd`, `benchmarks/run_ast_benchmarks.py:115`) on a machine that also has a resolvable `sg`/`ast-grep` binary (it benchmarks against it) — on such a machine, `tg run` now prefers the wrapper by default, so a fresh run may be timing "tg shelling out to sg" against "sg directly" rather than native-tree-sitter vs sg. **Positioning caveat (do not drop):** ast-grep is the structural-search BASELINE and `tg run` is "a useful validated AST slice, not a blanket ast-grep replacement" (`AGENTS.md`; the docs-governance tests ban an "ast-grep replacement" framing). Never let these ratios feed a "tg beats ast-grep" narrative. Re-run the script before citing a fresher number — cold-process, single-pass, and confirm which backend actually served the request (the `profile-guided-byte-identical-optimization` skill's warm-vs-cold measurement discipline applies here too: a cached/warm run of either binary understates its true per-invocation cost). ### 2a. The deep symbol-graph tier — `lang_registry` (added 2026-07-23) The two backends above answer "match/rewrite an AST pattern" (`tg run`/`tg scan`). A separate, third tier answers a different question — "what are this repo's symbols, and how do they import/call each other" (`tg orient`/`tg defs`/`tg source`/`tg imports`/`tg callers`/ `tg blast-radius`/the `tg agent` capsule) — and is unrelated code: no shared availability gate and no shared routing function with `AstBackend`/`AstGrepWrapperBackend` above. This tier's single source of truth is `lang_registry.py` (`src/tensor_grep/cli/lang_registry.py`): a frozen `LanguageSpec` dataclass registry answering "which languages does the symbol graph support, and which callable implements each extraction stage for each" (module docstring). `repo_map.py` calls `lang_registry.register_language(LanguageSpec(...))` once per language. **Do not hand-derive this count or its tier split — this repo's memory records the split being stated WRONG three separate times** (a hand-count landing on 4/10 with go demoted, a grep for a string that happened to also match `=None` landing on 8/10, and an invented third "unresolved" tier built from a field's absence). The authoritative source is the product's own descriptor function, run it rather than trust any number written here: ```bash python -c "import sys;sys.path.insert(0,'src');from tensor_grep.cli import repo_map as r;print(r._symbol_navigation_descriptor())" # -> parser-backed-refs-callers:c-cpp-csharp-go-java-javascript-php-python-rust-typescript # +foundational-defs-imports-only: # (as of Task 10E, the final wave of the top-10 language-support campaign: every registered # language is parser-backed, the foundational-defs-imports-only segment is EMPTY -- it is still # always emitted, never omitted, so the descriptor's shape never changes) ``` cross-checked against `grep -c "lang_registry.register_language(" src/tensor_grep/cli/repo_map.py`, which returns **10**. **All 10 of the top-10 languages** by TIOBE-Jul-2026/Stack-Overflow-2025/ GitHub-Octoverse-2025 consensus ranking (Python, JavaScript, TypeScript, Java, C#, C++, C, Go, Rust, PHP) are registered — there is no unregistered language on this list and nothing deliberately deferred. **There is also no third tier.** The 10 have been evenly parser-backed since the campaign's final waves -- PR #927 (Task 10A), the Task 10B C# wave, the Task 10C PHP wave, the Task 10D C wave, and the Task 10E C++ wave: **all ten parser-backed** languages with refs/callers support (c, cpp, csharp, go, java, javascript, php, python, rust, typescript; every one is in-file only -- cross-file caller confirmation still falls back to the text prefilter pending a package/source-root resolver) and **zero foundational** languages (defs+imports-only tier EMPTY). The descriptor's own docstring says this outright: *"As of Task 10E (C++, the final wave of the top-10 language-support campaign) this tier is EMPTY -- every registered language carries a real `references_and_calls` extractor"* (`repo_map.py:590-597`; the 2026-08-04 "C++ joins the symbol graph as a FOUNDATIONAL-TIER language" registration comment is pre-wave history). The real open backlog item here is therefore **cross-file caller confirmation for all ten languages** -- not registering or upgrading any language; the tier structure is final at 10/0. A language's callables live either as older helpers defined directly in `repo_map.py` (python needs no external grammar at all -- it parses with the stdlib `ast` module; rust's `_rust_*` helpers predate/mirror that inline style; java's `_java_*` helpers still hold its defs/imports extraction inline, but its references-and-calls extraction moved to `lang_java.py`, mirroring the module-shaped languages below) or in a newer, self-contained per-language module mirroring `lang_go.py` (`lang_go.py`, `lang_php.py`, `lang_csharp.py`, `lang_c.py`, `lang_cpp.py`, `lang_java.py` -- each importing nothing from `repo_map.py`, to avoid an import cycle). `LanguageSpec` does not care which shape a language's callables take, only that they exist, so both are equally contract-consistent — do not assume "inline" means "old" or "module" means "new" from the shape alone; check the registration date. **The fail-closed default matters more here than in section 2 above.** `LanguageSpec. provenance_when_missing` defaults to `"regex-heuristic"` (`lang_registry.py:89`) — the original four languages fall back differently when their grammar is missing: python's `provenance_when_missing="python-ast"` (grep that exact string in `repo_map.py` -- was `:6011`, now `:6294`, no external grammar to miss at all); javascript/typescript keep the inherited `"regex-heuristic"` default (grep each `register_language(` call and read the block — there is no separate literal string to grep for on those two, since it is the field's own default), while rust now sets it EXPLICITLY (grep `provenance_when_missing="regex-heuristic"` in `repo_map.py`). Every language added since the original four (go/java/php/csharp, and both c/cpp -- foundational-tier at registration, since promoted in Tasks 10D/10E) instead sets `provenance_when_missing="grammar-missing"` explicitly on its own `register_language(...)` call (count with `grep -c "^\s*provenance_when_missing=" src/tensor_grep/cli/repo_map.py` — **8 field settings** as of 2026-08-12 (was 6 as of 2026-08-09): the six post-registry `grammar-missing` languages plus python's `python-ast` plus rust's now-explicit `regex-heuristic`; the literal also appears in explanatory comments, so match the field shape `^\s*provenance_when_missing=` to count settings rather than raw hits) and ships **no regex fallback at all**: a grammar-absent file for one of these six returns `([], [])` from `_imports_and_symbols_for_path` (grep `def _imports_and_symbols_for_path` in `repo_map.py` -- was `:6626`, now `:6627`) rather than silently degrading. That flag is consumed by `_language_coverage_gaps_for_universe` (grep `def _language_coverage_gaps_for_universe` in `repo_map.py` -- was `:8461`, now `:8478`; the check itself is the literal `if spec.provenance_when_missing not in {"regex-heuristic", "heuristic"}:` -- grep that string rather than trust a sub-line offset, was `:8019`, now `:8515`), which turns it into an honest, labeled `resolution_gaps` entry instead of a silent empty result — the Backend Fail-Closed Contract's "treat a zero as UNKNOWN, never as a silently proven zero" rule, applied at the language-registration layer instead of the backend layer. This is a deliberate per-language precision/recall tradeoff, not an oversight; see `tensor-grep-change-control` if you are adding an 11th language and need the full seam checklist rather than the theory. **A concrete consequence of this design worth knowing (ties back to §3/§4's ranking theme below):** `_target_language_for_path` (grep `def _target_language_for_path` in `repo_map.py` -- was `:7850`, now `:7867`) feeds the `tg agent` capsule's query-language-vs-target-language confidence cap (`agent_capsule.py`). Its own in-repo comment calls each new-language branch the "MOST-FORGOTTEN seam" — miss it, and the capsule never learns the new language exists as a candidate target, so it can silently misfire (e.g. reporting "no target language" for a C# file instead of `primary_target_language == "csharp"`) with no error, just a quietly wrong answer. Same failure shape as the ranking weak points below: a missing registration doesn't crash, it degrades a downstream signal invisibly. **Positioning (ties back to the ast-grep discussion above):** **text search = any language** (`rg` passthrough, no tg-side language awareness at all); **structural scan/rewrite = 26 languages** (`tg run`/`tg scan`, via the ast-grep CLI this section describes — `_SUPPORTED_AST_LANGUAGES`, `ast_backend.py:76-103`, `get_supported_languages()` at `:128`); **deep symbol-graph = 10 languages, split across two tiers** (this subsection -- 10 parser-backed refs/callers + 0 foundational defs/imports-only: the foundational tier is EMPTY since the Task 10E C++ wave, per the `_symbol_navigation_descriptor()` derivation above). tg is `rg` (text) + ast-grep (structural) + a symbol/retrieval/capsule LAYER on top of that — not "a faster grep," and the three tiers do NOT share a language-support number, so check which tier a coverage claim is actually about before citing it. --- ## 3. BM25 / IDF — two different scorers in this repo, only one has IDF **Term rarity weighting in one sentence:** IDF (inverse document frequency) down-weights terms that appear in most documents (useless for discriminating relevance) and up-weights terms that appear in few (highly discriminating) — BM25 combines that with term-frequency saturation (diminishing returns for repeating a term) and document-length normalization. tg has **two independent ranking surfaces**, and only one of them actually implements IDF: 1. **`retrieval_bm25.py`** (`src/tensor_grep/core/retrieval_bm25.py`) — a real Okapi BM25 index (`k1=1.5`, `b=0.75`, standard defaults; `DEFAULT_K1`/`DEFAULT_B`, lines 18-19), full IDF term `math.log(1.0 + (n - freq + 0.5) / (freq + 0.5))` (line 44). This backs `tg search --rank` (alias `--bm25`; grep `"--bm25"` in `main.py` -- was `:7135`, now `:7395`) via `reranker.py::rerank_by_bm25`, which chunks matched files and re-sorts matches by the BM25 score of the chunk containing each match (`reranker.py:162-214`, stable sort at line 203) — a stable sort, so ties keep original grep order. 2. **The `tg orient` / capsule symbol-ranking family** (`src/tensor_grep/cli/repo_map.py`) — **a flat presence-count stack, no IDF anywhere in it.** Three layered pieces, not one function — do not conflate them: - `_score_text_terms` (grep `def _score_text_terms` in `repo_map.py` -- was `:7912`, now `:8189`) — the primitive: counts term hits in a haystack, no rarity weighting. - `_score_symbol` (grep `def _score_symbol` in `repo_map.py` -- was `:8177`, now `:8194`) — the actual per-symbol composite scorer, and the thing that produces `symbol["score"]`: name-match (`_score_text_terms` on the symbol name, `x3` weight) + kind-match + file-path score (`_score_file_path`, grep `def _score_file_path` in `repo_map.py` -- was `:8100`, now `:8117`), plus two additive heuristics shipped for task #254 (the CEO deep-research #251 steal / A7): a **+1 word-boundary bonus** (`_symbol_name_exact_boundary_bonus`, grep `def _symbol_name_exact_boundary_bonus` in `repo_map.py` -- was `:8159`, now `:8176`; fires when a query term longer than 3 chars matches a clean token in `split_terms(symbol_name)` rather than only a raw substring) and a **`_TEST_SHADOW_PENALTY = 2`** demotion (grep `_TEST_SHADOW_PENALTY` in `repo_map.py` -- was `:7661`, now `:8157`, floored at 0 in `_score_symbol`) that sinks a test-file hit below a same-named non-test definition instead of letting it compete on equal footing. Both are additive refinements to *order among already-matching candidates* — neither changes *which* symbols match, and neither adds IDF. (The two "as of 2026-07-27" hedges this bullet list used to carry had already drifted a further 17 lines by the very next pass — a dated hedge does not stop a line number from rotting, it just makes the rot look supervised, so this document tags no citation with a date anymore; see the note beside the AST-routing citation in §2 for the same lesson applied there.) - `_symbol_rank_key` (grep `def _symbol_rank_key` in `repo_map.py` -- was `:8044`, now `:8061`) — the final sort key, called as `scored_symbols.sort(key=_symbol_rank_key)` (grep that exact call in `repo_map.py` -- was `:8685`, now `:9181`). Its 7-tuple is `(query_match_rank, -score, kind-is-function?, -span_length, file, line, name)`. The **first** field, `query_match_rank`, is a query-relevance bucket (0 = `exact_query_match`, 1 = `bridge_query_match`, 2 = `covered_query_match`, 3 = none) evaluated **before** the flat `_score_symbol` score — so a query-name-match bucket dominates the flat count, it doesn't lose to it. The **final** tie-break field is `str(symbol.name)`, **not** a file-path string. This whole stack feeds `tg orient`'s symbol ranking and the `tg agent` capsule's target selection; the top-N candidate cap is `ranked_symbols[: max(max_symbols, 8)]` (grep that exact expression in `repo_map.py` -- was `:13187,13364`, now `:13683,13860`). **Why this is still a known weak point, just a narrower one than it used to be:** `_score_symbol` still has no IDF, so two symbols in the same `query_match_rank` bucket can still tie on the flat `score` — but `query_match_rank` being evaluated first, plus four more tie-break fields (kind, span length, file, line) sitting ahead of `name`, make an *unrelated* corpus change flipping the final pick considerably less likely — and less exactly reproducible — than it was when this was first found. The original incident (receipt in project memory `tensor-grep-idf-ranking-fragility-2026-06-29`): a corpus change with **zero call-graph edge** to the query — an unrelated file added elsewhere in the repo — shifted which candidate won a flat-score tie and flipped the capsule's primary target, including flipping "ask before editing" (`ambiguity=tie_requires_confirmation, ask_user=True`) to "confidently pick a target" (`ask_user=False`) with no code-level connection an agent could see via `tg callers` (PR #302). That incident predates both the `query_match_rank` first-field and the `_score_symbol` heuristics documented above, so do not assume today's tuple shape reproduces it step-for-step on a fresh repro attempt — but the underlying hazard (a flat, no-IDF score can tie, and a tie still falls through several non-relevance fields before `name`) is real and unresolved. It is covered by a **degrade-to-ask safety floor**, not a ranking fix: if the post-tie primary target is still an unrequested "marker" helper, `agent_capsule.py`'s `_primary_target_is_unrequested_marker_helper` (now `agent_capsule_targets.py` -- find it: `grep -rn "^def _primary_target_is_unrequested_marker_helper" src/tensor_grep/cli/`) forces `ask_user=True` rather than silently auto-picking it. The flat no-IDF scorer family itself remains deferred debt — do not assume it has been fixed just because the unsafe *consequence* was mitigated, and do not mistake the `query_match_rank` bucketing or the #254/A7 heuristics for an IDF fix: they are relevance refinements layered on the same rarity-blind foundation, not term-rarity weighting. If you are reviewing a PR that touches ranking-feature tests: an edit that reddens a live-repo ranking assertion is not automatically a "brittle test" to relax — inspect whether the actual tie/ask/confidence behavior degraded before deciding. See `tensor-grep-idf-ranking-fragility-2026-06-29` in project memory for the full incident writeup, and `tensor-grep-change-control` for the review gate. --- ## 4. PageRank / centrality — and why `tg orient` deliberately does NOT use it tg has a real, hand-rolled **personalized PageRank** implementation over the reverse-import graph: `_personalized_reverse_import_pagerank` (`src/tensor_grep/cli/repo_map.py:9174` — re-derive with: grep -n '_personalized_reverse_import_pagerank' src/tensor_grep/cli/repo_map.py) — damping factor `alpha=0.85` (the standard Google PageRank default), `12` power-iteration steps, a personalization vector seeded uniformly over up to `_GRAPH_PAGERANK_SEED_FILE_LIMIT = 64` query- relevant files (grep `_GRAPH_PAGERANK_SEED_FILE_LIMIT` in `repo_map.py` -- was `:319`, now `:335`), teleporting back to those seeds rather than to a uniform distribution. This feeds descriptive-query file ranking (`graph-centrality` reason) inside `repo_map`/capsule/edit- plan retrieval — pure Python, no `networkx` dependency (unlike Aider's repo-map, which uses `networkx`'s PageRank over the full import graph — an external comparison, not yet documented in this repo's `docs/tool_comparison.md`, which currently makes no Aider/networkx/PageRank claim). **`tg orient`'s "central files" list is explicitly NOT PageRank — it's a composite of import in-degree plus symbol density, both capped** (`src/tensor_grep/cli/orient_capsule.py:694`, `_central_files_from_map`; docstring: *"Rank source files by import in-degree (foundational = imported-by-many); top-N with symbols"*). The rationale for avoiding raw reverse-import PageRank here — that a personalized PageRank seeded by all files ranks IMPORTERS above the imported, which is backwards for "show me the core files" — is no longer stated as a verbatim code comment at this location; do not quote it as a literal in-repo string without re-finding it. The underlying design choice is still real and still worth citing conceptually: **personalized PageRank answers "what's relevant to this specific seed set", while in-degree answers "what does the whole repo depend on"** — different questions, and `orient` wants the second (foundational files a newcomer should read first). The current implementation additionally caps fan-in (`_CENTRAL_FAN_IN_CAP = 12`) and symbol density (`_CENTRAL_SYMBOL_DENSITY_CAP = 25`, both `orient_capsule.py:45-46`) so a single widely-imported data-sink file or one giant file can't dominate the ranking on its own — a refinement on top of plain in-degree, not a switch to PageRank. If you're adding a new "show me the important files" feature, pick deliberately between personalized-PageRank and in-degree-based centrality; don't default to whichever is already imported in the module you're editing. --- ## 5. Trigram index — `--index` / warm-cache acceleration A trigram index maps every 3-byte substring ("trigram") appearing in the corpus to the list of files containing it (a postings list); a query first extracts the trigrams it must contain, intersects their postings lists to get a small file candidate set, then only regex-scans those files instead of every file in the corpus. tg's implementation: `TrigramIndex` struct, `rust_core/src/index.rs:151`, 3-byte keys (`FileTrigramHits = Vec<([u8; 3], u32)>`, line 30), binary bincode serialize/deserialize. **Safety property:** when a pattern has no extractable required literal (e.g. `.*` or an alternation with no common substring), the index cannot safely prefilter — the code falls back to a full scan "so the index never introduces false negatives" (`index.rs:1609`). A trigram index is a *prefilter*, never a source of truth by itself; getting this fallback wrong would mean silently missing real matches, which is strictly worse than being slow. **Compatibility gate for warm-index auto-routing** (`docs/routing_policy.md` line 52): the router only auto-routes to `TrigramIndex` on a warm, non-stale cache when the query is index-compatible — `pattern >= 3 bytes`, and none of `-v`, `-C`, `--max-count`, `-w`, `-g` are present. Below 3 bytes there's no trigram to extract; the other flags change result *shape* in ways the index path doesn't (yet) replicate. `--index` (explicit) is priority 1 in the router's decision tree regardless of staleness (`routing_policy.md` lines 36, 48). --- ## 6. PyO3 + the GIL The **GIL** (Global Interpreter Lock) serializes Python bytecode execution to one thread at a time. When Rust code called via PyO3 (the Rust↔Python FFI binding library tg uses for its native extension) does CPU-bound work with no Python API calls in the loop, holding the GIL for that whole stretch blocks *every other Python thread* pointlessly — the fix is to explicitly release it around the pure-Rust portion. tg does this correctly at the mmap/newline-scan boundary: `py.detach(|| { ... })` wraps the call to `create_arrow_string_array_from_mmap` in both `read_mmap_to_arrow` and its chunked sibling (`rust_core/src/lib.rs:32,55`) — comment: *"Release the GIL while we map the file and scan for newlines"* (line 31). `py.detach` is PyO3's current API name for what older PyO3 code (and most tutorials/training data) calls `py.allow_threads` — same mechanism, releases the GIL for the closure and reacquires it after. **The module pin is a live scar, not a style choice.** `#[pymodule(gil_used = true)]` (`rust_core/src/lib.rs:353`) intentionally opts back into the classic (non-free-threaded) GIL model. The comment explains why: a prior attempt to ship `gil_used = false` (free-threaded Python, #266) broke Linux `agent-readiness` in CI, and because that PR's CI run was cancelled by a force-push, it merged without ever going green — re-enabling free-threading requires a **full green CI run on Linux extension load** first, not just a local pass (`lib.rs:350-352`). **Settled battle, don't re-propose:** FFI (moving directory-walk/file-scanning work into the PyO3 extension "for speed") was tried and reverted — the FFI call overhead measured higher than native CPython directory scanning for that workload. See `tensor-grep-failure-archaeology` before proposing a PyO3 rewrite of a hot path; benchmark first (`tensor-grep-benchmark-and-proof-toolkit`). --- ## 7. MCP — the protocol, and its argv-injection surface **MCP (Model Context Protocol)** is the tool-calling protocol that lets an LLM agent invoke structured "tools" (typed functions with a JSON schema) exposed by a server process. tg's MCP server is built on the official Python `mcp` SDK's `FastMCP` class: `from mcp.server.fastmcp import FastMCP` (`src/tensor_grep/cli/mcp_server.py:20`, still current) / `mcp = FastMCP("tensor-grep")` (grep that exact call in the same file -- was `:120`, now `:189`), a ~7700-line module (`mcp_server.py`) — it has grown substantially (from ~4500 lines) on unrelated feature work since this doc's baseline; re-check the line count before citing it as a "small module" argument in a review. **The domain risk that matters here:** an MCP tool handler takes LLM-supplied parameters (a search `pattern`, a file `path`, a rewrite `replacement`) and forwards them into a subprocess argv to invoke the native `tg` binary. If a parameter value happens to start with `-`, and the argv builder doesn't end option-parsing first, the "data" is reinterpreted as a flag — this is CWE-88 (argument injection), the same class as the MCP-276 CVE. **List-argv subprocess calls (no `shell=True`) already block shell injection; they do NOT block flag injection** — that needs an explicit `--` sentinel before user-controlled positionals. tg's rewrite/index-search command builders do this: `command.extend(["--", pattern, path])` (grep that exact call in `mcp_server.py` -- was `:1306`, now `:1375`) and the parallel index-search builder, `_build_index_search_command` (grep `def _build_index_search_command` in `mcp_server.py` -- was `:1362-1372`, now `:1379-1390`). If you add a new MCP tool that shells out with user-controlled string values, this is the pattern to copy — and the gap to check for if you don't see it. **The native rg-passthrough path-sentinel gap is now FIXED, not op …(truncated)