Tensor Grep Debugging Playbook

tensor-grep Debugging Playbook

A symptom-first runbook for the recurring ways tg (or its CI/release pipeline) breaks. Every row below was a real, previously-diagnosed failure in this repo — not a hypothetical. The single biggest time-waster on record is theorizing from a stack trace instead of reading the structured failure first: a README rewrite once cost 4 CI cycles because the team guessed at causes from tracebacks instead of decoding which CI check actually failed (AGENTS.md). Do not repeat that.

When NOT to use this skill

This is a triage skill (symptom → cause → experiment → fix pointer), not a how-to or a history book. Reach for a sibling instead when:

If your symptom isn't in the table below, it's probably not covered here — check tensor-grep-failure-archaeology for a prior occurrence before assuming it's novel.

Jargon, defined once

• Front door — the entry point argv must pass through to be routed correctly. tg's Python front door is tensor_grep.cli.bootstrap:main_entry; it intercepts plain-text searches and forwards them to rg before the Typer app sees argv. CliRunner in tests calls the Typer app directly and bypasses this front door, so a routing bug can be invisible to green unit tests.
• Fail-closed — on a real failure, raise/error instead of silently returning a clean-looking empty result or swapping to an engine that can't honor the requested semantics.
• Push-race — two main-bound merges overlapping so the second git push origin main from an in-flight semantic-release job is rejected non-fast-forward.
• Registration site — one of several places a new command/flag/route must be added; missing one makes it silently misroute instead of erroring loudly.
• argv/flag injection (CWE-88) — a user- or LLM-controlled value that begins with - gets parsed by a subprocess's own argument parser as a flag instead of as data, even when the parent process used list-argv (shell=False), which only stops shell injection.
• Capture surface — the mechanism a test reads a command's output through. CliRunner's result.stdout/result.output captures only in-process writes (typer.echo/click.echo during .invoke()); pytest's capfd captures at the OS file-descriptor level, which is the only way to see output written by a real exec'd subprocess. They are not interchangeable, and using the wrong one doesn't error — it silently reads back empty. See §9.

Triage table

1. CI red — decode the structured check first

Do not read the traceback and start theorizing. Identify which named job/check failed, then read only that job's failed-step log.

gh pr checks <PR-number>              # which named check(s) actually failed
gh run view <run-id> --json jobs      # confirm the job name, e.g. "Semantic Release", "test-python"
gh run view <run-id> --log-failed     # only the failed step's log — not the whole 20-minute run

Why this matters here specifically: this repo's CI enforces far more than tests — formatting, typing, cross-platform behavior, release-workflow contracts, package-manager contracts, and artifact/version parity all block the same pipeline (docs/CI_PIPELINE.md). A registration mismatch (new command/flag missing one of its sites) fails the blocking registration-completeness gate, which is a different job than test-python, and reading a Python traceback from the wrong job wastes a cycle. Registration sites and rules live in tensor-grep-change-control; the checker itself is src/tensor_grep/core/registration_check.py (check_group_smart, extract_members), exercised by tests/unit/test_registration_check.py.

Known real incident: a README rewrite broke ~14 governance tests and a separate agent-readiness release-blocker gate; 4 CI cycles were wasted because the team theorized from tracebacks instead of reading which check failed first (root cause was two unrelated layers: a missing ast-grep CLI dependency, and uv run re-syncing away the [dev] tree-sitter extra). Decode the check name before touching code.

A second, more subtle version of the same trap: a red test-python job whose failure signature (JSONDecodeError from an empty captured string) looks like a routing regression but is actually a stale test fixture — the test was reading the wrong capture stream after a delegation-routing change moved the command from a subprocess path to an in-process one. Reading the traceback alone sends you looking for a routing bug that doesn't exist; the fix pointer is §9, not a backend change.

If the failing check is the Semantic Release job specifically, go to §2, not here.

2. Release did not publish (push-race)

The real publish step is the Semantic Release job inside .github/workflows/ci.yml, which compiles native assets before publishing (~6 minutes) — that whole window is a race window where a second merge to main can knock out the first run's final push.

Discriminating experiment:

gh run view <run-id> --json jobs                 # find the "Semantic Release" job's run/conclusion
gh run view <run-id> --log-failed                 # read its failed step only

A line reading ! [rejected] main -> main is the push-race signature. Do not panic-rerun — the failure self-heals on the next push-to-main (version is derived from git tags, not the failed run's state). Full mechanism, the v1.17.23/#318/#319 receipt, and the one-merge-per-tick discipline to prevent recurrence: tensor-grep-release-and-positioning §1.5 / tensor-grep-failure-archaeology Battle 6.

A SECOND, different release-failure branch does NOT self-heal — read the job conclusion before picking a recovery, don't assume every "release didn't publish" is a push-race:

Rapid-window batch-merge is a third, benign shape — don't misdiagnose it as either of the above. Several independently-green PRs merging ~15-20s apart can show an intermediate cancelled or rejected-push run that looks alarming in isolation, but is fine IF the LAST run in the sequence completes and publishes (receipt: v1.93.0/#703-706, runs 29890576036 rejected-only / 29890612228 published). See tensor-grep-change-control Part 7 (C-batch) before treating a mid-sequence cancelled conclusion as a failure needing recovery at all.

3. Search hangs/slow

tg search does NOT have one timeout contract — it has three distinct outcomes depending on which route executes the search (verified 2026-08-12 against bootstrap.py + rust_core/src/rg_passthrough.rs; SUPERSEDES this section's earlier wording that claimed BOTH routes fail fast at 60s/exit 124):

The 60s default was lowered from 600s specifically because ripgrep does GB/s and a >60s search means something pathological is being scanned (an unexcluded huge/index directory), not a legitimately slow query. On the PYTHON route's timeout, the child is killed and the process exits 124 with a stderr hint to scope the search or raise the timeout (src/tensor_grep/cli/bootstrap.py, backward-compat shim path and the primary Popen/_terminate_child path both return 124 — re-verify with grep -n "return 124" src/tensor_grep/cli/bootstrap.py; was :1020/:1063-1071, then :1269/:1320, now :1353/:1404 — line numbers drift every release). The native route's ceiling applies ONLY when the walk is implicit (path_was_implicit); a user-scoped native search skips the ceiling and spawns rg directly into the unbounded-wait arm.

Discriminating experiment: check the exit code — it names the route. 124 = the PYTHON bootstrap timeout fired (not a crash). 2 with the "broad root scan refused" marker = the native implicit-walk ceiling fired BEFORE any rg spawn. NO exit at all (a real hang) = the unbounded spawned-rg arm — Ctrl-C is legitimate there, and scoping the search is the fix. Compare a scoped vs. unscoped run:

tg search PATTERN                 # unscoped over a large/whole repo — can hit the 60s wall
tg search PATTERN src/            # scoped — typically <1s

Root cause when it fires on a legitimately-sized repo: tg's own index/state directories (.tensor-grep/, _tg_refs/, .tg_semantic_index/) and vendored corpora (e.g. benchmarks/external_repos/) are not excluded from an unscoped walk, so searching from the repo root walks tg's own indices too.

Fix / workaround: always scope searches to a path, glob, or file type. Raise TG_RG_TIMEOUT_SECONDS (or TG_SUBPROCESS_TIMEOUT_SECONDS for non-search subprocess calls) only for a genuinely huge monorepo — do not raise it to paper over an unscoped-walk problem. A trigram-hybrid index is the tracked structural fix; own-dir excludes alone were tried and did not fully resolve full-tree speed. Full env-var reference: tensor-grep-config-and-flags.

Related, known limitation — tg inventory --deadline on a pathological workspace-union tree: tg inventory --deadline is a different command from tg search but shares the same root-cause class as the hang above (an unbounded directory read), and normally bounds cleanly per project (truncates at N files, stamps truncation_cause = "deadline" — build_inventory has since moved out of main.py into its own module; re-verify with grep -n 'truncation_cause = "deadline"' src/tensor_grep/cli/inventory.py; was in main.py -- the :8404/:8420 pins pointed INSIDE a --deadline option block deleted by the 2026-08-23 de-duplication, so they have no successor; now inventory.py:318). On a PATHOLOGICAL workspace-union tree — many unrelated repos flattened under one huge root, not a single normal project — it can still blow its deadline: the shared walker _iter_repo_files (re-verify with grep -n "def _iter_repo_files" src/tensor_grep/cli/repo_map.py; was :1143, now :1144) reads an entire huge directory's entries in one non-lazy list(os.scandir(normalized_root)) call inside that same function (re-verify with grep -n "list(os.scandir(normalized_root))" src/tensor_grep/cli/repo_map.py; was :1009, now :1172 — the def itself barely moved but this internal call drifted much further as the function's docstring grew) before its own per-file deadline check gets a chance to run, so one abnormally large subdirectory can exceed the deadline before the mid-walk check fires even once. This is a KNOWN, accepted, low-priority edge (rare shape; verified against a real 300k+-file multi-project workspace) — not worth a load-bearing lazy-scandir rewrite. Don't re-diagnose it as a new bug; if the SAME deadline-blown symptom shows up on a normal single-project repo (not a workspace union), that IS a regression and should be treated as a new incident, not this one.

4. Silent-empty result (fail-closed contract)

Every ComputeBackend must raise BackendExecutionError on a real failure — never return a clean 0-match SearchResult, and never silently swap to an engine that cannot preserve the requested semantics (src/tensor_grep/backends/base.py:6-14). This has been violated repeatedly; the recurring anti-pattern is a bare except Exception: that returns empty or falls through to a different engine. A context tool reporting a trustworthy-looking "no matches" when the real answer is "the backend crashed" is the one failure this repo treats as unacceptable (AGENTS.md, "Backend Fail-Closed Contract").

Discriminating experiment: run the same pattern/path directly through rg and compare. If rg finds matches but tg reports zero, suspect a swallowed backend error, not a real no-match. Then inspect --json output for routing_reason / fallback_reason — a populated fallback_reason means a visible, legitimate degraded path (e.g. CyBERT provider unavailable); an absent one on a result you believe is wrong means look for a silent swap. Current, still-live example of the correct visible-degrade shape (v1.77.0, #189): tg find's JSON carries rank_fallback_reason when the dense leg degrades to BM25-only (the semantic extra or model is unavailable) — a legitimate, fully-supported result, distinguishable from a real backend failure (which instead raises BackendExecutionError -> exit 2, per tensor-grep-run-and-operate §11c). If you see a tg find result with NEITHER rank_fallback_reason set NOR a nonzero exit on a run you expected the dense leg to participate in, that is the silent-swap bug this section targets, not a normal degrade.

Ground-truth example of the correct pattern (src/tensor_grep/backends/rust_backend.py:260-278): a PCRE2 search that fails inside the native ripgrep bridge raises BackendExecutionError and explicitly refuses to fall back to an engine that doesn't implement PCRE2 semantics — it does NOT silently re-run the pattern through the Python-regex engine (which would return wrong matches, not zero matches, but the principle is the same: don't swap engines invisibly for a semantics-changing flag). Contrast with a legitimate degraded fallback (limit/sort flags the Python fallback can't honor), which instead sets a visible bridge_fallback_reason on the result.

Fix pointer: if you find a bare except Exception: return SearchResult(...) (or similar) in a backend, that is the bug class. Fail closed for any flag/contract the fallback cannot preserve (raise, don't swap); if a degraded fallback is legitimate, set fallback_reason + routing_reason so JSON/CLI consumers can tell degraded output from real output. Deep architecture of this contract: tensor-grep-architecture-contract.

5. Argv/flag injection

A list-argv subprocess call (shell=False) stops shell injection but not flag injection: a value beginning with - is parsed by the child's own option parser as a flag. This is CWE-88 — the same class behind live MCP-server CVEs (CVE-2026-5058 aws-mcp-server, CVE-2026-23744, CVE-2026-30623 Anthropic MCP SDK) — and it matters here because MCP tool handlers forward LLM-controlled parameter values straight into tg/rg/git subprocess argv.

Discriminating experiment:

tg search -- --looks-like-a-flag PATH     # with -- sentinel: treated as pattern data
tg search --looks-like-a-flag PATH        # without: rg/tg's own parser errors on the "flag"

Run the same probe through any code path that builds subprocess argv from a pattern/path/replacement value (MCP tool handlers, rewrite commands) — a value beginning with - should error or be treated as data, never silently change tg's own behavior.

Fixed reference implementation (src/tensor_grep/cli/mcp_server.py, _build_rewrite_command / _build_index_search_command — re-verify with grep -n "def _build_rewrite_command\|def _build_index_search_command" src/tensor_grep/cli/mcp_server.py; was :1259/:1310, now :1328/:1379, +69 each): a -- end-of-options sentinel is inserted before the user-controlled pattern/path positionals, with an inline comment explaining why.

Round-4 native-passthrough gap — RESOLVED, do not reopen (verified current at v1.49.3). rust_core/src/rg_passthrough.rs appending paths directly with no -- sentinel (a directory literally named -l parsed by rg as the -l/files-with-matches flag instead of a path) was fixed in #326 (v1.17.26), silently regressed by a later refactor, then restored in #370 (v1.28.1) as the extracted, unit-tested ripgrep_operand_args helper (rust_core/src/rg_passthrough.rs — see the grep below; was :581-600, now :584-603) — the sentinel is now pushed unconditionally before the path loop whenever !args.paths.is_empty(). Patterns going through -e were never affected (-e consumes the next token as its value regardless of a leading -); only bare path positionals were ever at risk, and that risk is now closed. Verify with grep -n "fn ripgrep_operand_args" -A 20 rust_core/src/rg_passthrough.rs before relying on this — do not trust the naive grep -n "for path in &args.paths" re-check below, it still matches (the loop still exists, just now after the unconditional sentinel push) and would misread as "still open" if you stop at the grep hit without reading the surrounding function.

Caveats worth knowing before you conclude a builder is safe: -- protects only what comes after it — a positional placed before -- is still injectable; it does not gate --flag=VALUE forms; and not every binary honors -- the same way, so dogfood the real binary rather than trusting the argv list alone. None of {validate the value, list-argv, -- sentinel} alone is complete — they layer.

6. Mock-green-real-dead

A test can pass because it mocked the exact boundary that was actually broken — a monkeypatched function, or a Python-side stub standing in for the compiled PyO3 extension. This has happened for real: mock-based FFI tests were green while the real Rust bridge was dead (it dropped every forwarded flag and silently fell back to the Python engine) — the dead-passthrough bug and the missing-flag bug compounded, because the bridge call itself never got exercised (AGENTS.md, "Local Dev Gotchas").

Discriminating experiment: does the test import/patch tensor_grep.rust_core (or its Python wrapper RustCoreBackend, the try: from tensor_grep.rust_core import RustBackend as NativeRustBackend / HAVE_RUST block in src/tensor_grep/backends/rust_backend.py — re-verify with grep -n "HAVE_RUST" src/tensor_grep/backends/rust_backend.py; was :28-33, now :9-14), or does it patch something around that boundary? If a test replaces bootstrap.run_subprocess or stubs RustCoreBackend.inner, it is validating call shape, not that the real extension does the right thing.

uv run python -c "from tensor_grep.backends.rust_backend import HAVE_RUST; print(HAVE_RUST)"
# then, separately, exercise the REAL installed binary end to end (not CliRunner):
tg search --pcre2 'foo(bar)?' src/            # confirm the flag actually reaches rg with real semantics

Same principle one layer up: CliRunner invokes the Typer app directly and bypasses the tensor_grep.cli.bootstrap:main_entry front door entirely, so a routing bug in the bootstrap layer is invisible to CliRunner-based tests no matter how many pass. After any change to a search flag, a command, or the FFI boundary, dogfood the installed published binary with the harness at scripts/dogfood/ (Dockerfile + dogfood_features.py) rather than trusting unit tests alone (AGENTS.md, "Dogfood the Real Binary, Not CliRunner"). See dogfood-the-shipped-artifact (global skill) for the full post-release procedure.

7. Dependency-cap silent downgrade

An upper-bound pin (e.g. typer<0.26) can silently downgrade the entire package on a newer Python if no release in that range is compatible with it — pip/uv resolve the whole install down to a stale version with no error, because requires-python>=X has no upper bound to catch the mismatch. Receipt: on Python 3.14, uv tool install tensor-grep with an unsatisfiable typer<0.25 range resolved to a stale 1.13.35 instead of erroring. Current pin, chosen to thread both constraints (pyproject.toml:560-566):

typer>=0.12,<0.26

The comment there (pyproject.toml:560-565) explains why the cap can't simply be dropped: typer 0.26 removed click.testing.CliRunner inheritance, breaking CliRunner.isolated_filesystem() which ~49 tests rely on.

Discriminating experiment:

pip index versions tensor-grep                 # what SHOULD be installable
uvx --refresh-package tensor-grep --from tensor-grep==<expected-version> tg --version

If a fresh install on a new Python resolves to an old tg --version, do not assume requires-python is wrong — grep pyproject.toml for < upper bounds on typer, click, pydantic, or other transitive deps first; that is the class of bug this was.

8. Ranking flip

The agent capsule's primary-target candidate selection relies on three scoring helpers in repo_map.py — _score_symbol, _score_import_entry, _score_file_source_terms — plus score_term_overlap (src/tensor_grep/core/retrieval_lexical.py:15), which _score_file_source_terms calls. Re-verify their positions before citing one; they do NOT move together, only relative to each other:

grep -n "def _score_symbol\|def _score_import_entry\|def _score_file_source_terms" src/tensor_grep/cli/repo_map.py
grep -n "score_term_overlap(" src/tensor_grep/cli/repo_map.py

_score_symbol used to sit after the other two (:8211 vs :7725/:7732, with the call site at :7737) and now sits before them (:8194 vs :8221/:8228, call site now :8233) — a relative reordering, not a uniform shift, so don't assume a fixed offset holds between any two of these four line numbers. Together they implement a flat, no-IDF set-membership scorer plus a hard top-N candidate cap — an acknowledged, not-yet-fixed weak point. A small, unrelated corpus change can flip which candidate wins a near-tie, and that flip is invisible to the call graph (nothing "broke" in the traditional sense — the ranking function just picked a different winner). This produced a real incident: an unrelated GPU-code change flipped the agent capsule's top pick from "tied, ask the user" to "confidently pick the wrong marker/no-op function" with zero call-graph signal.

Note: tg search --rank (rerank_by_bm25(), src/tensor_grep/core/reranker.py) and local semantic search (src/tensor_grep/core/semantic_index.py) both route through Bm25Index (src/tensor_grep/core/retrieval_bm25.py) — a real Okapi BM25 scorer with IDF, term-frequency saturation, and length normalization. They are a different, IDF-weighted scorer and are not known to share this

…(truncated)