Code Search And Retrieval Reference

Code Search & Retrieval Reference

The domain-theory pack a mid-level engineer (or a model working cold) usually lacks, narrowed to only the slice that governs tensor-grep's actual behavior. Every claim below cites the tg file that uses it — read that file before relying on the claim in a review or a fix, because code drifts and this document does not update itself. Verified against the repo as of 2026-07-08, v1.49.3; §9's tg find addition and the new §10 (query-shape classification) verified 2026-07-16, v1.78.1; §3's _score_symbol/_symbol_rank_key breakdown re-verified and corrected 2026-07-22, v1.93.2. Every file:line citation in the document re-grepped against origin/main, §2's AST-routing description corrected, and new §2a (the lang_registry symbol-graph tier) added, 2026-07-23, v1.95.0 — see the dated note at the end of "Provenance and maintenance" for what changed and why. 2026-08-01, v1.101.27: §2a's language-tier claim corrected (all 10 top-10 languages are registered, not 8; there is no unregistered-C/C++ gap) and every drifted file:line citation in the document converted to a grep <symbol> instruction carrying a was -> now receipt — see the final dated note in "Provenance and maintenance" for the full list and for why a "verified as of DATE" tag on a line number does not stop it from rotting.

When NOT to use this skill (use a sibling instead)

This skill explains why a subsystem behaves the way it does; it does not tell you how to run, fix, or ship it.

1. ripgrep internals — the cold-path baseline

rg is tg's raw-text-search comparator and, for most default tg search invocations, the actual execution engine underneath (RipgrepBackend, src/tensor_grep/backends/ripgrep_backend.py; native Rust passthrough in rust_core/src/rg_passthrough.rs). Getting rg's edge semantics wrong means either a silent wrong-answer or a parity-test false green.

Exit codes are not binary. ripgrep uses three, and tg's contract is to match rg's exit code exactly, including the non-obvious one — and exit 2 itself is not simply pass-or-raise, because of the partial-results contract (see tensor-grep-architecture-contract for the full SearchResult.result_incomplete story, added #341):

Do not describe exit-2 as unconditionally "raise BackendExecutionError" — that was true before #341 (round-4 slice 3) and is no longer true; a partial exit-2 with real matches is now a kept result, not a failure. tests/e2e/test_rg_parity_edges.py::test_rg_exit_code_edges_match is parametrized exactly on the fatal-failure boundary (grep ids=\["match", "no-match", "parse-error", in the same file -- was line 149, now 169) and asserts tg.returncode == rg.returncode for every case (grep def _assert_same_rg_behavior -- was line 81, now 96) — a regex ( (unbalanced paren, nothing parsed) must still make tg exit 2, same as rg. Partial-parse-with-matches is covered separately by tests/unit/test_rg_exit2_partial.py.

PCRE2 is a different engine, not a flag. rg's default matcher is the Rust regex crate — linear time, but no lookaround, no backreferences. --pcre2/-P switches to libpcre2, which supports both but requires the pattern to be valid UTF-8 (it transcodes). tg detects PCRE2 support with a real smoke test — build help output contains --pcre2/PCRE2, then actually run rg -P "a(?=b)" -V and check returncode == 0 (grep def supports_pcre2 in ripgrep_backend.py -- was :27-51, now :54-77). This matters because of the fail-closed contract: --pcre2 routed through an engine that cannot honor PCRE2 semantics must raise, never silently execute as a plain-regex search that returns wrong (or merely different) matches (src/tensor_grep/backends/base.py:7-14, BackendExecutionError; grep **Fail closed** in AGENTS.md -- was line 444, now :2096). A prior incident shipped exactly this bug — a broad except Exception: pass around the Rust passthrough silently ran --pcre2 through the non-PCRE2 Python-regex engine — fixed in v1.17.17/18 (see tensor-grep-change-control, grep ## Part 4 in that skill's SKILL.md -- was lines 125-134, now :287-311 -- Backend fail-closed contract).

Binary detection is NUL-byte sniffing, and it changes exit codes. rg's default binary heuristic scans early file bytes for a \0; on a hit, the file is treated as binary and searched under the binary-skip policy unless -a/--text is passed (rg_contract.py row "text", public_flags: ("-a", "--text")). The parity fixture builds exactly this case — binary_path.write_bytes(b"needle\0binary tail\n") (grep that exact line in test_rg_parity_edges.py -- was :41-43, now :44) — and the "binary-skip" parametrization (grep ids=\["match", "no-match", "parse-error", -- was line 149, now 169) asserts tg's exit code matches rg's on a NUL-containing file, not just its stdout.

-u/-uu/-uuu are not blind passthrough here. Upstream, each additional -u widens scope (-u = --no-ignore, -uu = --no-ignore --hidden, -uuu = --no-ignore --hidden --binary). tg's Python front door specifically detects any -u* flag (or --unrestricted, or an explicit no-ignore/hidden flag) as a request for unrestricted scanning and routes it through a broad-root safety guard (grep def _search_args_request_unrestricted_generated_scan in bootstrap.py -- was :581-590, now :726-732). This exists because of a real v1.13.1 incident: an unguarded broad-root unrestricted scan could recurse into node_modules/.git/multi-project workspace roots. If you're adding a new flag that widens scan scope, check whether it needs to join this guard's flag set — a missed case is a silent safety regression, not just a slow query.

-- and -e matter for argv safety, not just POSIX correctness. -- ends option parsing so a user- or LLM-supplied pattern beginning with - cannot be reinterpreted as a flag (CWE-88 / the MCP-276 CVE class). tg's MCP tool handlers build subprocess argv with an explicit -- sentinel before positionals for exactly this reason: command.extend(["--", pattern, path]) (grep that exact call in src/tensor_grep/cli/mcp_server.py -- was :1306, now :1375, comment two lines above: "round-3 security: end options before the user-controlled positionals so a pattern beginning with - cannot be parsed by the native binary as a flag"). Note the narrower scope of this fix: it blocks flag injection via a missing --, not shell injection (list-argv subprocess calls already block shell injection). See tensor-grep-change-control before touching any subprocess argv builder.

BOM handling is a real, previously-broken seam. UTF-8 BOM bytes at the start of a file/scenario JSON broke PowerShell-generated fixtures until scenario loading switched to utf-8-sig (docs/PAPER.md:840). AST rewrite's batch-apply path explicitly preserves BOM/CRLF through atomic writes (docs/harness_api.md:700, "batch apply reuses the same atomic-write, BOM/CRLF preservation, binary-skip, and stale-file protections as single rewrites").

Open / candidate — verify before relying on these, no in-repo citation exists yet. Two upstream ripgrep quirks worth probing against tg's passthrough the next time someone does a round-4-style edge sweep: (a) --multiline --pcre2 --json reportedly emits a single match with two submatches rather than two matches (upstream ripgrep issue tracker; exact issue number unverified against this repo — re-check before citing a number); (b) rg -c (count mode) reportedly omits files whose content contains a NUL byte from the count entirely, rather than counting them normally. Neither has a tensor-grep test fixture as of this writing — treat as candidate, not fact, until one exists.

2. ast-grep + tree-sitter — structural search

Two backends implement structural (AST-aware, not text-regex) search and rewrite, and the router picks between them per query:

tree-sitter parses source into a concrete syntax tree; a metavariable like $FUNC or the "capture the rest" form $$$ARGS is ast-grep/tg's pattern-matching primitive over that tree (e.g. def $FUNC($$$ARGS): matches any Python function definition, binding $FUNC and $$$ARGS).

Corrected — the routing default is the OPPOSITE of what this section previously said. The real routing decision lives in _select_ast_backend_for_pattern in src/tensor_grep/cli/ast_workflows.py (grep def _select_ast_backend_for_pattern in ast_workflows.py -- the main.py copy this instruction used to point at (was :6737, then :6915) is now a THIN FORWARDING SHIM onto the ast_workflows implementation; grep "Thin forwarding shim onto" src/tensor_grep/cli/main.py finds it, and its docstring records the drift that forced the collapse -- the old hand-maintained duplicate silently dropped the requires_ast_grep_wrapper fail-closed guard. A dated hedge does not stop a line number from rotting, it just makes the rot look supervised, so no citation in this document carries a date anymore), and its own comment is unambiguous: "Prefer the ast-grep wrapper whenever it is available: it is the stable, results-defining backend for BOTH pattern kinds. The native tree-sitter AstBackend uses a DIFFERENT DSL and returns DIFFERENT results, so it must not be silently preferred ... Native-as-CPU-default is task #141. Native is reached ONLY as the ast-grep-absent fallback for native patterns." (grep "Prefer the ast-grep wrapper" in ast_workflows.py -- was :6692-6697 in main.py, then :6952-6957; now ast_workflows.py:1231-1232 (grep "Prefer the ast-grep wrapper")). Concretely: the wrapper availability check runs first (if _check_backend_available("AstGrepWrapperBackend"): backend = _get_cached_backend("AstGrepWrapperBackend") -- grep _check_backend_available("AstGrepWrapperBackend") in ast_workflows.py; the old if ast_wrapper.is_available(): backend = ast_wrapper shape this entry used to cite in main.py, was :6698, then :6958, is gone), unconditionally, regardless of GPU/CUDA or pattern shape; native AstBackend is reached only when the wrapper is unavailable AND the pattern qualifies as "native" (base_config.ast_prefer_native and is_native_ast_language(base_config.lang) — the native-capable language set is narrower than the wrapper's, _NATIVE_AST_LANGUAGES = ("python", "javascript", "typescript", "tsx", "rust"), ast_backend.py:104). This applies to AST search, --rewrite planning, --apply, --diff, and batch rewrite flows alike. docs/routing_policy.md's own "AST commands" section is itself stale here (grep ## AST commands in that doc -- was :107-113, now :156) — it still describes a torch-geometric/CUDA-style gate — trust the code cited above, not that doc's prose, until it is refreshed.

Practical corollary (updated): on almost any real dev or CI box, ast-grep/sg being installed is now the thing that decides the backend, not GPU presence — with ast-grep installed (the common case, since it's how most people actually got tg run working), tg run/AST calls use the CLI sidecar regardless of CUDA. This used to be a CUDA story: an earlier "AST probe" bug in CI traced back to AstBackend being GPU-gated (see tensor-grep-docs-and-writing Part 6, and project memory tensor-grep-readme-release-blocker-2026-06-25, for that incident) — but the GPU gate itself is gone (#542 above), so don't cite CUDA absence as the reason native is skipped anymore; cite wrapper availability instead. Don't assume native AST speed numbers apply unless you've confirmed the wrapper is unavailable (or --lang/pattern shape forced native) on the box you're measuring.

Measured (not marketing) ratios, benchmarks/run_ast_benchmarks.py / run_ast_multilang_benchmarks.py (docs/benchmarks.md "ast-grep vs tensor-grep AST mode"): single- query tg 0.116s vs sg 0.151s (0.770x); multi-language ratios Python 0.722x, JavaScript 0.800x, TypeScript 0.726x, Rust 0.715x — tg ahead of sg on all four when the native path is actually reachable. Given the routing correction above, treat these specific numbers as unverified-current: run_ast_benchmarks.py invokes the real tg binary end-to-end (build_tg_ast_benchmark_cmd, benchmarks/run_ast_benchmarks.py:115) on a machine that also has a resolvable sg/ast-grep binary (it benchmarks against it) — on such a machine, tg run now prefers the wrapper by default, so a fresh run may be timing "tg shelling out to sg" against "sg directly" rather than native-tree-sitter vs sg. Positioning caveat (do not drop): ast-grep is the structural-search BASELINE and tg run is "a useful validated AST slice, not a blanket ast-grep replacement" (AGENTS.md; the docs-governance tests ban an "ast-grep replacement" framing). Never let these ratios feed a "tg beats ast-grep" narrative. Re-run the script before citing a fresher number — cold-process, single-pass, and confirm which backend actually served the request (the profile-guided-byte-identical-optimization skill's warm-vs-cold measurement discipline applies here too: a cached/warm run of either binary understates its true per-invocation cost).

2a. The deep symbol-graph tier — lang_registry (added 2026-07-23)

The two backends above answer "match/rewrite an AST pattern" (tg run/tg scan). A separate, third tier answers a different question — "what are this repo's symbols, and how do they import/call each other" (tg orient/tg defs/tg source/tg imports/tg callers/ tg blast-radius/the tg agent capsule) — and is unrelated code: no shared availability gate and no shared routing function with AstBackend/AstGrepWrapperBackend above.

This tier's single source of truth is lang_registry.py (src/tensor_grep/cli/lang_registry.py): a frozen LanguageSpec dataclass registry answering "which languages does the symbol graph support, and which callable implements each extraction stage for each" (module docstring). repo_map.py calls lang_registry.register_language(LanguageSpec(...)) once per language.

Do not hand-derive this count or its tier split — this repo's memory records the split being stated WRONG three separate times (a hand-count landing on 4/10 with go demoted, a grep for a string that happened to also match =None landing on 8/10, and an invented third "unresolved" tier built from a field's absence). The authoritative source is the product's own descriptor function, run it rather than trust any number written here:

python -c "import sys;sys.path.insert(0,'src');from tensor_grep.cli import repo_map as r;print(r._symbol_navigation_descriptor())"
# -> parser-backed-refs-callers:c-cpp-csharp-go-java-javascript-php-python-rust-typescript
#    +foundational-defs-imports-only:
# (as of Task 10E, the final wave of the top-10 language-support campaign: every registered
# language is parser-backed, the foundational-defs-imports-only segment is EMPTY -- it is still
# always emitted, never omitted, so the descriptor's shape never changes)

cross-checked against grep -c "lang_registry.register_language(" src/tensor_grep/cli/repo_map.py, which returns 10. All 10 of the top-10 languages by TIOBE-Jul-2026/Stack-Overflow-2025/ GitHub-Octoverse-2025 consensus ranking (Python, JavaScript, TypeScript, Java, C#, C++, C, Go, Rust, PHP) are registered — there is no unregistered language on this list and nothing deliberately deferred. There is also no third tier. The 10 have been evenly parser-backed since the campaign's final waves -- PR #927 (Task 10A), the Task 10B C# wave, the Task 10C PHP wave, the Task 10D C wave, and the Task 10E C++ wave: all ten parser-backed languages with refs/callers support (c, cpp, csharp, go, java, javascript, php, python, rust, typescript; every one is in-file only -- cross-file caller confirmation still falls back to the text prefilter pending a package/source-root resolver) and zero foundational languages (defs+imports-only tier EMPTY). The descriptor's own docstring says this outright: "As of Task 10E (C++, the final wave of the top-10 language-support campaign) this tier is EMPTY -- every registered language carries a real references_and_calls extractor" (repo_map.py:590-597; the 2026-08-04 "C++ joins the symbol graph as a FOUNDATIONAL-TIER language" registration comment is pre-wave history). The real open backlog item here is therefore cross-file caller confirmation for all ten languages -- not registering or upgrading any language; the tier structure is final at 10/0. A language's callables live either as older helpers defined directly in repo_map.py (python needs no external grammar at all -- it parses with the stdlib ast module; rust's _rust_* helpers predate/mirror that inline style; java's _java_* helpers still hold its defs/imports extraction inline, but its references-and-calls extraction moved to lang_java.py, mirroring the module-shaped languages below) or in a newer, self-contained per-language module mirroring lang_go.py (lang_go.py, lang_php.py, lang_csharp.py, lang_c.py, lang_cpp.py, lang_java.py -- each importing nothing from repo_map.py, to avoid an import cycle). LanguageSpec does not care which shape a language's callables take, only that they exist, so both are equally contract-consistent — do not assume "inline" means "old" or "module" means "new" from the shape alone; check the registration date.

The fail-closed default matters more here than in section 2 above. LanguageSpec. provenance_when_missing defaults to "regex-heuristic" (lang_registry.py:89) — the original four languages fall back differently when their grammar is missing: python's provenance_when_missing="python-ast" (grep that exact string in repo_map.py -- was :6011, now :6294, no external grammar to miss at all); javascript/typescript keep the inherited "regex-heuristic" default (grep each register_language( call and read the block — there is no separate literal string to grep for on those two, since it is the field's own default), while rust now sets it EXPLICITLY (grep provenance_when_missing="regex-heuristic" in repo_map.py). Every language added since the original four (go/java/php/csharp, and both c/cpp -- foundational-tier at registration, since promoted in Tasks 10D/10E) instead sets provenance_when_missing="grammar-missing" explicitly on its own register_language(...) call (count with grep -c "^\s*provenance_when_missing=" src/tensor_grep/cli/repo_map.py — 8 field settings as of 2026-08-12 (was 6 as of 2026-08-09): the six post-registry grammar-missing languages plus python's python-ast plus rust's now-explicit regex-heuristic; the literal also appears in explanatory comments, so match the field shape ^\s*provenance_when_missing= to count settings rather than raw hits) and ships no regex fallback at all: a grammar-absent file for one of these six returns ([], []) from _imports_and_symbols_for_path (grep def _imports_and_symbols_for_path in repo_map.py -- was :6626, now :6627) rather than silently degrading. That flag is consumed by _language_coverage_gaps_for_universe (grep def _language_coverage_gaps_for_universe in repo_map.py -- was :8461, now :8478; the check itself is the literal if spec.provenance_when_missing not in {"regex-heuristic", "heuristic"}: -- grep that string rather than trust a sub-line offset, was :8019, now :8515), which turns it into an honest, labeled resolution_gaps entry instead of a silent empty result — the Backend Fail-Closed Contract's "treat a zero as UNKNOWN, never as a silently proven zero" rule, applied at the language-registration layer instead of the backend layer. This is a deliberate per-language precision/recall tradeoff, not an oversight; see tensor-grep-change-control if you are adding an 11th language and need the full seam checklist rather than the theory.

A concrete consequence of this design worth knowing (ties back to §3/§4's ranking theme below): _target_language_for_path (grep def _target_language_for_path in repo_map.py -- was :7850, now :7867) feeds the tg agent capsule's query-language-vs-target-language confidence cap (agent_capsule.py). Its own in-repo comment calls each new-language branch the "MOST-FORGOTTEN seam" — miss it, and the capsule never learns the new language exists as a candidate target, so it can silently misfire (e.g. reporting "no target language" for a C# file instead of primary_target_language == "csharp") with no error, just a quietly wrong answer. Same failure shape as the ranking weak points below: a missing registration doesn't crash, it degrades a downstream signal invisibly.

Positioning (ties back to the ast-grep discussion above): text search = any language (rg passthrough, no tg-side language awareness at all); structural scan/rewrite = 26 languages (tg run/tg scan, via the ast-grep CLI this section describes — _SUPPORTED_AST_LANGUAGES, ast_backend.py:76-103, get_supported_languages() at :128); deep symbol-graph = 10 languages, split across two tiers (this subsection -- 10 parser-backed refs/callers + 0 foundational defs/imports-only: the foundational tier is EMPTY since the Task 10E C++ wave, per the _symbol_navigation_descriptor() derivation above). tg is rg (text) + ast-grep (structural) + a symbol/retrieval/capsule LAYER on top of that — not "a faster grep," and the three tiers do NOT share a language-support number, so check which tier a coverage claim is actually about before citing it.

3. BM25 / IDF — two different scorers in this repo, only one has IDF

Term rarity weighting in one sentence: IDF (inverse document frequency) down-weights terms that appear in most documents (useless for discriminating relevance) and up-weights terms that appear in few (highly discriminating) — BM25 combines that with term-frequency saturation (diminishing returns for repeating a term) and document-length normalization.

tg has two independent ranking surfaces, and only one of them actually implements IDF:

1. retrieval_bm25.py (src/tensor_grep/core/retrieval_bm25.py) — a real Okapi BM25 index (k1=1.5, b=0.75, standard defaults; DEFAULT_K1/DEFAULT_B, lines 18-19), full IDF term math.log(1.0 + (n - freq + 0.5) / (freq + 0.5)) (line 44). This backs tg search --rank (alias --bm25; grep "--bm25" in main.py -- was :7135, now :7395) via reranker.py::rerank_by_bm25, which chunks matched files and re-sorts matches by the BM25 score of the chunk containing each match (reranker.py:162-214, stable sort at line 203) — a stable sort, so ties keep original grep order.

2. The tg orient / capsule symbol-ranking family (src/tensor_grep/cli/repo_map.py) — a flat presence-count stack, no IDF anywhere in it. Three layered pieces, not one function — do not conflate them:

   • _score_text_terms (grep def _score_text_terms in repo_map.py -- was :7912, now :8189) — the primitive: counts term hits in a haystack, no rarity weighting.

   • _score_symbol (grep def _score_symbol in repo_map.py -- was :8177, now :8194) — the actual per-symbol composite scorer, and the thing that produces symbol["score"]: name-match (_score_text_terms on the symbol name, x3 weight) + kind-match + file-path score (_score_file_path, grep def _score_file_path in repo_map.py -- was :8100, now :8117), plus two additive heuristics shipped for task #254 (the CEO deep-research #251 steal / A7): a +1 word-boundary bonus (_symbol_name_exact_boundary_bonus, grep def _symbol_name_exact_boundary_bonus in repo_map.py -- was :8159, now :8176; fires when a query term longer than 3 chars matches a clean token in split_terms(symbol_name) rather than only a raw substring) and a _TEST_SHADOW_PENALTY = 2 demotion (grep _TEST_SHADOW_PENALTY in repo_map.py -- was :7661, now :8157, floored at 0 in _score_symbol) that sinks a test-file hit below a same-named non-test definition instead of letting it compete on equal footing. Both are additive refinements to order among already-matching candidates — neither changes which symbols match, and neither adds IDF.

     (The two "as of 2026-07-27" hedges this bullet list used to carry had already drifted a further 17 lines by the very next pass — a dated hedge does not stop a line number from rotting, it just makes the rot look supervised, so this document tags no citation with a date anymore; see the note beside the AST-routing citation in §2 for the same lesson applied there.)

   • _symbol_rank_key (grep def _symbol_rank_key in repo_map.py -- was :8044, now :8061) — the final sort key, called as scored_symbols.sort(key=_symbol_rank_key) (grep that exact call in repo_map.py -- was :8685, now :9181). Its 7-tuple is (query_match_rank, -score, kind-is-function?, -span_length, file, line, name). The first field, query_match_rank, is a query-relevance bucket (0 = exact_query_match, 1 = bridge_query_match, 2 = covered_query_match, 3 = none) evaluated before the flat _score_symbol score — so a query-name-match bucket dominates the flat count, it doesn't lose to it. The final tie-break field is str(symbol.name), not a file-path string.

   This whole stack feeds tg orient's symbol ranking and the tg agent capsule's target selection; the top-N candidate cap is ranked_symbols[: max(max_symbols, 8)] (grep that exact expression in repo_map.py -- was :13187,13364, now :13683,13860).

Why this is still a known weak point, just a narrower one than it used to be: _score_symbol still has no IDF, so two symbols in the same query_match_rank bucket can still tie on the flat score — but query_match_rank being evaluated first, plus four more tie-break fields (kind, span length, file, line) sitting ahead of name, make an unrelated corpus change flipping the final pick considerably less likely — and less exactly reproducible — than it was when this was first found. The original incident (receipt in project memory tensor-grep-idf-ranking-fragility-2026-06-29): a corpus change with zero call-graph edge to the query — an unrelated file added elsewhere in the repo — shifted which candidate won a flat-score tie and flipped the capsule's primary target, including flipping "ask before editing" (ambiguity=tie_requires_confirmation, ask_user=True) to "confidently pick a target" (ask_user=False) with no code-level connection an agent could see via tg callers (PR #302). That incident predates both the query_match_rank first-field and the _score_symbol heuristics documented above, so do not assume today's tuple shape reproduces it step-for-step on a fresh repro attempt — but the underlying hazard (a flat, no-IDF score can tie, and a tie still falls through several non-relevance fields before name) is real and unresolved. It is covered by a degrade-to-ask safety floor, not a ranking fix: if the post-tie primary target is still an unrequested "marker" helper, agent_capsule.py's _primary_target_is_unrequested_marker_helper (now agent_capsule_targets.py -- find it: grep -rn "^def _primary_target_is_unrequested_marker_helper" src/tensor_grep/cli/) forces ask_user=True rather than silently auto-picking it. The flat no-IDF scorer family itself remains deferred debt — do not assume it has been fixed just because the unsafe consequence was mitigated, and do not mistake the query_match_rank bucketing or the #254/A7 heuristics for an IDF fix: they are relevance refinements layered on the same rarity-blind foundation, not term-rarity weighting.

If you are reviewing a PR that touches ranking-feature tests: an edit that reddens a live-repo ranking assertion is not automatically a "brittle test" to relax — inspect whether the actual tie/ask/confidence behavior degraded before deciding. See tensor-grep-idf-ranking-fragility-2026-06-29 in project memory for the full incident writeup, and tensor-grep-change-control for the review gate.

4. PageRank / centrality — and why tg orient deliberately does NOT use it

tg has a real, hand-rolled personalized PageRank implementation over the reverse-import graph: _personalized_reverse_import_pagerank (src/tensor_grep/cli/repo_map.py:9174 — re-derive with: grep -n '_personalized_reverse_import_pagerank' src/tensor_grep/cli/repo_map.py) — damping factor alpha=0.85 (the standard Google PageRank default), 12 power-iteration steps, a personalization vector seeded uniformly over up to _GRAPH_PAGERANK_SEED_FILE_LIMIT = 64 query- relevant files (grep _GRAPH_PAGERANK_SEED_FILE_LIMIT in repo_map.py -- was :319, now :335), teleporting back to those seeds rather than to a uniform distribution. This feeds descriptive-query file ranking (graph-centrality reason) inside repo_map/capsule/edit- plan retrieval — pure Python, no networkx dependency (unlike Aider's repo-map, which uses networkx's PageRank over the full import graph — an external comparison, not yet documented in this repo's docs/tool_comparison.md, which currently makes no Aider/networkx/PageRank claim).

tg orient's "central files" list is explicitly NOT PageRank — it's a composite of import in-degree plus symbol density, both capped (src/tensor_grep/cli/orient_capsule.py:694, _central_files_from_map; docstring: "Rank source files by import in-degree (foundational = imported-by-many); top-N with symbols"). The rationale for avoiding raw reverse-import PageRank here — that a personalized PageRank seeded by all files ranks IMPORTERS above the imported, which is backwards for "show me the core files" — is no longer stated as a verbatim code comment at this location; do not quote it as a literal in-repo string without re-finding it. The underlying design choice is still real and still worth citing conceptually: personalized PageRank answers "what's relevant to this specific seed set", while in-degree answers "what does the whole repo depend on" — different questions, and orient wants the second (foundational files a newcomer should read first). The current implementation additionally caps fan-in (_CENTRAL_FAN_IN_CAP = 12) and symbol density (_CENTRAL_SYMBOL_DENSITY_CAP = 25, both orient_capsule.py:45-46) so a single widely-imported data-sink file or one giant file can't dominate the ranking on its own — a refinement on top of plain in-degree, not a switch to PageRank. If you're adding a new "show me the important files" feature, pick deliberately between personalized-PageRank and in-degree-based centrality; don't default to whichever is already imported in the module you're editing.

5. Trigram index — --index / warm-cache acceleration

A trigram index maps every 3-byte substring ("trigram") appearing in the corpus to the list of files containing it (a postings list); a query first extracts the trigrams it must contain, intersects their postings lists to get a small file candidate set, then only regex-scans those files instead of every file in the corpus. tg's implementation: TrigramIndex struct, rust_core/src/index.rs:151, 3-byte keys (FileTrigramHits = Vec<([u8; 3], u32)>, line 30), binary bincode serialize/deserialize.

Safety property: when a pattern has no extractable required literal (e.g. .* or an alternation with no common substring), the index cannot safely prefilter — the code falls back to a full scan "so the index never introduces false negatives" (index.rs:1609). A trigram index is a prefilter, never a source of truth by itself; getting this fallback wrong would mean silently missing real matches, which is strictly worse than being slow.

Compatibility gate for warm-index auto-routing (docs/routing_policy.md line 52): the router only auto-routes to TrigramIndex on a warm, non-stale cache when the query is index-compatible — pattern >= 3 bytes, and none of -v, -C, --max-count, -w, -g are present. Below 3 bytes there's no trigram to extract; the other flags change result shape in ways the index path doesn't (yet) replicate. --index (explicit) is priority 1 in the router's decision tree regardless of staleness (routing_policy.md lines 36, 48).

6. PyO3 + the GIL

The GIL (Global Interpreter Lock) serializes Python bytecode execution to one thread at a time. When Rust code called via PyO3 (the Rust↔Python FFI binding library tg uses for its native extension) does CPU-bound work with no Python API calls in the loop, holding the GIL for that whole stretch blocks every other Python thread pointlessly — the fix is to explicitly release it around the pure-Rust portion.

tg does this correctly at the mmap/newline-scan boundary: py.detach(|| { ... }) wraps the call to create_arrow_string_array_from_mmap in both read_mmap_to_arrow and its chunked sibling (rust_core/src/lib.rs:32,55) — comment: "Release the GIL while we map the file and scan for newlines" (line 31). py.detach is PyO3's current API name for what older PyO3 code (and most tutorials/training data) calls py.allow_threads — same mechanism, releases the GIL for the closure and reacquires it after.

The module pin is a live scar, not a style choice. #[pymodule(gil_used = true)] (rust_core/src/lib.rs:353) intentionally opts back into the classic (non-free-threaded) GIL model. The comment explains why: a prior attempt to ship gil_used = false (free-threaded Python, #266) broke Linux agent-readiness in CI, and because that PR's CI run was cancelled by a force-push, it merged without ever going green — re-enabling free-threading requires a full green CI run on Linux extension load first, not just a local pass (lib.rs:350-352).

Settled battle, don't re-propose: FFI (moving directory-walk/file-scanning work into the PyO3 extension "for speed") was tried and reverted — the FFI call overhead measured higher than native CPython directory scanning for that workload. See tensor-grep-failure-archaeology before proposing a PyO3 rewrite of a hot path; benchmark first (tensor-grep-benchmark-and-proof-toolkit).

7. MCP — the protocol, and its argv-injection surface

MCP (Model Context Protocol) is the tool-calling protocol that lets an LLM agent invoke structured "tools" (typed functions with a JSON schema) exposed by a server process. tg's MCP server is built on the official Python mcp SDK's FastMCP class: from mcp.server.fastmcp import FastMCP (src/tensor_grep/cli/mcp_server.py:20, still current) / mcp = FastMCP("tensor-grep") (grep that exact call in the same file -- was :120, now :189), a ~7700-line module (mcp_server.py) — it has grown substantially (from ~4500 lines) on unrelated feature work since this doc's baseline; re-check the line count before citing it as a "small module" argument in a review.

The domain risk that matters here: an MCP tool handler takes LLM-supplied parameters (a search pattern, a file path, a rewrite replacement) and forwards them into a subprocess argv to invoke the native tg binary. If a parameter value happens to start with -, and the argv builder doesn't end option-parsing first, the "data" is reinterpreted as a flag — this is CWE-88 (argument injection), the same class as the MCP-276 CVE. List-argv subprocess calls (no shell=True) already block shell injection; they do NOT block flag injection — that needs an explicit -- sentinel before user-controlled positionals. tg's rewrite/index-search command builders do this: command.extend(["--", pattern, path]) (grep that exact call in mcp_server.py -- was :1306, now :1375) and the parallel index-search builder, _build_index_search_command (grep def _build_index_search_command in mcp_server.py -- was :1362-1372, now :1379-1390). If you add a new MCP tool that shells out with user-controlled string values, this is the pattern to copy — and the gap to check for if you don't see it.

**The native rg-passthrough path-sentinel gap is now FIXED, not op

…(truncated)