Tensor Grep Failure Archaeology

Tensor-Grep Failure Archaeology

A chronicle of settled battles in tensor-grep: expensive fights that already reached a verdict. Each entry is symptom -> root cause -> evidence -> status so a future engineer or model does not burn a day re-discovering the same wall.

Facts below were verified against the repo on 2026-07-02 at v1.17.25, with a second pass on 2026-07-03 at v1.19.3 (Battles 9-14), a third pass adding Battle 15 (release-tag-smoke), a fourth pass 2026-07-16 at v1.78.1 fixing the stale case_sensitive KNOWN_GAP claim in Battle 9 and adding Battle 16 (the tg find Opus-gate catches), a fifth pass 2026-07-22 at v1.93.2 adding Battles 17-22 (the CEO deep-research campaign's research retirements — cAST, dense int8, warm-session, GPU-for-search, the many-pattern dedup bug, and the "5/5 mirage" meta-lesson), and a sixth pass 2026-07-24 at v1.96.0 adding Battles 23-24 (the language-expansion campaign's stale onboarding-brief catch and its sequential multi-PR shared-file union-rebase discipline), a warm-hides-cold addendum to Battle 12, and re-pointing Battle 16's evidence at its post-squash commit hashes (501dc26/6d79945) plus fixing a stale AGENTS.md section-letter reference, and a seventh, same-day pass 2026-07-24 at v1.98.2 adding Battle 25 (the banked C function-pointer fix hypothesis, falsified before any code was written), and an eighth pass 2026-07-31 at v1.101.24 adding Battles 26-29 (the defaulted-scope disclosure census undercount, the anonymous --claim sentinel-id retention, the MaxSim rerank-extra deliberate hold, and the #868 GPU exit-2 investigation's two false-confirmation traps), a ninth pass 2026-08-01 at v1.101.27 adding Battle 30 (the -q false-zero shipped in the shared _build_cmd, #876/#880), and a tenth retention pass 2026-08-12 that marked Battle 28 RETIRED (task F10, per the retrieval_late.py module header), Battle 29 SETTLED/RETIRED (commit 50595ef — the exit-2 rule itself was retired), scoped Battle 20's no-crossover claim to single-pattern + fair public baselines, corrected Battle 21's #694 description (the guard pins the CORRECT behavior, not the bug) and Battle 23's lang_java.py claim, and reconciled Battle 22's steal-list count to its actual 6-candidate list. Re-verify anything load-bearing with the commands in Provenance and maintenance before you act on it.

When to use this skill

Load this before you spend effort on any of these, because each has already been fought:

• Proposing to move directory traversal / file walking into Rust via a PyO3 extension "for speed".
• Re-enabling PyO3 free-threading (gil_used = false, #[pyclass(frozen)]).
• Adding a self-test or health check that runs tg search … --json ….
• Tightening a dependency upper cap (typer<X, click<X, pydantic<X, …).
• Diagnosing a release that "didn't publish" or a green-CI-then-no-PyPI gap.
• Blaming a ranking / capsule / semantic-search behavior change on IDF, or "fixing" it by widening a candidate pool.
• Trusting a green mock or FFI test as proof that a bridge works.
• Reacting to a wall of red governance tests after a README / docs change.
• Chasing a reported latency "regression" by guessing the hot path instead of profiling the actual slow command at scale.
• Shipping a doc-drift / precision-heuristic feature because its fixture tests are green.
• Adding a "differs-from-default" runtime gate to decide native-delegation refusal.
• Reading capfd/fd-level captured output from a CliRunner-invoked command.
• Micro-optimizing a hot loop (e.g. skip-unless-needed) without checking every consumer of the skipped value first.
• Gating a second sequential merge-watcher on "did the tag change since I started" instead of an absolute condition.
• Re-proposing cAST structural chunking as the default chunker — evaluated and REJECTED (Battle 17).
• Re-proposing dense-embedding int8/binary/PCA compression for speed — evaluated and DEFERRED, it's slower in numpy (Battle 18).
• Proposing a warm-session/daemon search-index shortcut ("just serve search from the daemon's cached state") — the daemon holds a symbol map, not a search index; this is a big refactor, not a quick win (Battle 19).
• Proposing GPU-for-search, or describing the shipped GPU kernel as PFAC/Aho-Corasick — it is a brute-force byte-compare with no single-pattern crossover at any scale vs the fair public baselines (Battle 20); many-fixed-pattern native CUDA remains only the local candidate lane.
• Hitting a many-pattern -e/-f count that looks off — a live native aho-corasick dedup over-count bug is already diagnosed and guarded, not yet root-fixed (Battle 21).
• Trusting an unverified "cheap win" from a paper or research steal-list without checking it against the real code first — every one of the 6 recent steals came back negative/big-refactor/secondary-path once verified (Battle 22, the meta-lesson).
• Assuming a "clone language X's extractor" onboarding brief is still accurate without checking which module is the CURRENT template — a brief that says "mirror inline _rust_*" is stale once the codebase has grown a lang_registry + lang_go.py-style module (Battle 23).
• Trusting a conflict-free git rebase across several PRs that all touch the same shared registration file (a suffix dict, an extras list, a lockfile) as proof the merged file is still complete (Battle 24).
• Trusting a cause-emitter census count ("N of N covered") that was reached by reasoning one emitter transitively covers another, instead of counting each one directly — the defaulted-scope disclosure census undercounted by one (Battle 26).
• Re-proposing an auto-derived agent-id for tg ledger --claim to remove the explicit --agent-id requirement — evaluated and REJECTED, it silently reproduces #845 through the ledger's own overlap suppression (Battle 27).
• Treating MaxSim reranking's absence after tg install-dense as a CUJ-coverage gap — MaxSim late-rerank is RETIRED 2026-08-05 (task F10) as a validated dead end, not a paused hold or a coverage gap (Battle 28).
• Chasing a GPU-path exit code 2 as a plain regression, or trusting a control/repro that reproduces the failing output, before confirming which exact symbol it exercised and whether that symbol's job runs in the real CI path (Battle 29).
• Adding a flag that suppresses or alters rg's output (e.g. -q) to the shared _build_cmd argv builder instead of search_passthrough — three of its four consumers PARSE stdout, and suppression reads as a false no-match, not a quieter one (Battle 30).

When NOT to use this skill (use a sibling instead)

This skill is read-only history. It does NOT authorize a change. Reopening any settled battle still goes through tensor-grep-change-control (council-verify -> dry-run -> conscious flag-flip). Nothing here routes around that.

How to read an entry

Every entry has four fields. Trust the Evidence column over the prose; if the evidence no longer matches the code, the battle may have moved — flag it, do not silently act.

• Symptom — what you would observe that tempts the re-fight.
• Root cause — the mechanism, stated precisely (the naive guess is usually wrong).
• Evidence — the commit / PR / file:symbol that proves it.
• Status — SETTLED (do not re-fight without change-control) or OPEN (known debt).

Battle 1 — PyO3 for directory walking (FFI boundary loses to os.walk)

Rule: FFI is the speed path only when it avoids the per-item boundary entirely — i.e. the embedded tg.exe route that maps files with memmap2 + rayon and never yields per-path into Python (docs/PAPER.md §3.6). A PyO3 extension that returns a large iterator of small Python objects will lose to CPython C code. Do not re-propose "just wrap ignore in PyO3" for the walk.

Battle 2 — PyO3 free-threading (gil_used=false) broke Linux, blocked all releases

Rule: A merge whose CI was cancelled by a force-push is not verified. Two independent traps compounded here (Linux-only import break + ungated merge). Do not re-flip free-threading casually, and never let a force-push swallow the gating CI run.

Battle 3 — the --json fork-bomb (mutual native<->Python delegation)

Rule: Never add a health check / self-test that runs tg search … --json … — that is the exact shape that reopens C3. If you touch the native<->Python delegation path, run tests/unit/test_launcher_no_respawn.py and dogfood the real binary, not CliRunner.

Battle 4 — README rewrite: 14 red governance tests + 4 wasted CI cycles

Rule (the load-bearing lesson): Decode the structured failing check FIRST. Run gh run view <id> --log-failed (or read the specific failing test's assertion) and fix that check. Do not theorize a root cause from a traceback and start reverting unrelated things — that is what cost 4 cycles. This same "decode first" rule is why Battle 6's push-race is diagnosable in one step.

Battle 5 — dependency upper-cap silently downgraded the whole install on Python 3.14

Rule: When a new Python version yields a stale install, suspect a transitive dependency upper cap (typer / click / pydantic), not requires-python. An upper cap that is unsatisfiable on a newer Python degrades silently — always validate an unpinned install on the newest supported Python end-to-end before assuming a cap is safe. Tightening any <X cap is a change-control event (tensor-grep-change-control).

Battle 6 — release "didn't publish": the push-race (non-fast-forward)

Recovery — do NOT panic-rerun. The failure self-heals: the next push-to-main re-runs Semantic Release, and because the version is derived from git tags (not the failed run's state), it recomputes the correct next version and covers the orphaned commit. Just confirm that next run's Semantic Release job succeeds. Diagnose with the structured result first: gh run view <id> --json jobs -> find Semantic Release -> gh run view <id> --log-failed. A ! [rejected] main -> main line is the push-race signature; anything else is a different problem.

Battle 7 — the IDF-ranking-fragility flip (the IDF root-cause guess was DISPROVEN)

Rule: Do not ship your first root-cause guess for a ranking flip (IDF was the wrong guess here). Ranking surfaces can silently degrade a safety behavior on corpus change — treat any capsule/--rank/semantic change as safety-relevant, keep the degrade-to-ask floor, and gate scorer changes on a measured benchmark (tensor-grep-benchmark-and-proof-toolkit), never a clean-looking diff. No speed/quality claim without numbers vs the accepted baseline.

Battle 8 — the broken-oracle trap: green mock/FFI tests while the real bridge was DEAD

Rule: Verify FFI/bridge changes against the REAL extension (a live maturin develop runtime call), never mocks alone. A green mock is a hypothesis, not proof (see the unwritten rule "never trust a self-report"). For the eval/grader version of this trap — a whole lane failing uniformly, a grader that passes an empty answer, pnpm.cmd exit-127 on Windows — load trustworthy-cuj-scoring, which enforces bidirectional oracle validation (a correct answer must PASS and a wrong/empty answer must FAIL before any batch is trusted).

Battle 9 — native delegation silently dropped --rank/--sort-files (+ the query_pattern landmine)

Rule: Same bug class as #336's -u/-uu no-op — a SearchConfig field that affects output is a silent-drop landmine unless it is forwarded to the native argv, added to the refuse-tuple, or explicitly gated. Adding a new SearchConfig field? Load tensor-grep-config-and-flags and run the coverage ratchet. Never propose a generic "differs-from-default" gate for delegation refusal — query_pattern makes it degenerate to "always refuse."

Battle 10 — the capfd capture-surface trap: a delegation-routing fix broke fd-level test assertions

Rule: A test that reads capfd against a CliRunner.invoke() result is implicitly asserting that the command delegated to a real subprocess (whose output crosses the OS fd boundary) rather than running in-process. Any change to native-delegation routing — what gets forwarded vs. refused — can silently flip which capture surface carries the real output. If you touch _can_delegate_to_native_tg_search or the refuse-tuple, run tests/integration/ locally, not just tests/unit/. This is the sibling of Battle 3's "dogfood the real binary, not CliRunner": here CliRunner is the right tool, but you must know which stream it captures for the path you just changed.

Battle 11 — MatchLine hashability regression, and a "free" micro-opt that wasn't

Rule: field(default=None, compare=False) is the fix shape for "add a field to a frozen dataclass that must stay hashable but holds an unhashable value type" (the offsets are a pure function of text+line, so excluding it from == is correct too). Before gating a per-match stash "because only formatter X consumes it," grep the actual consumers first — a test that breaks the instant you ship a profiler-motivated micro-opt is not an obstacle to route around; it is the consumer contract telling you the guess was wrong.

Battle 12 — the reported +33% blast-radius "regression" was noise; the council's own hot-path guess was also wrong

Rule: For a latency claim, the profiler is the oracle — not a diff-review council, and not even a first-pass profiler read: this hunt needed a second profiling pass (profile the real command at scale, with a call-count monkeypatch) before the true hotspot appeared, because the first profile ran on the wrong command entirely. Before accepting any "X% regression" report, reproduce it on the byte-identical code path across both versions before designing a fix — it may be noise (see also tensor-grep-benchmark-and-proof-toolkit). A pure, no-I/O helper function called repeatedly inside a graph/loop algorithm is a near-free @lru_cache candidate; check dict[str, set] → dict[str, frozenset] downstream type hints when the cached value is shared/iterated but never mutated in place.

Addendum (2026-07-24) — the mirror-image trap: a warm run can hide a real WIN too. The same "measure the actual thing, not a proxy" discipline cuts the other way: a warm end-to-end dogfood rerun measures whatever CACHED path the optimized function no longer sits on, so a real improvement can read as noise, or even a false regression, if the changed function only runs on the cold/ first-touch path. Two later optimizations — 9a2a01c ("merge 3 redundant ast.walk passes in _python_imports_and_symbols into one") and d2c1266 ("textual pre-check before per-candidate validation-test AST parse") — were verified with a single-pass, fresh-process microbenchmark over distinct inputs instead of a warm end-to-end dogfood rerun, specifically to avoid this trap; see profile-guided-byte-identical-optimization for the general technique. Same rule as this battle's own Rule paragraph, one layer deeper: the proxy trap this time is warm-cache reuse, not "profiled the wrong command."

Battle 13 — tg diff-docs: 17 green fixture tests, 20k+ false positives on the real corpus (deliberately deferred)

Rule: Fixture-green is not sufficient for a precision/heuristic feature. Before shipping anything that classifies or flags ("is this drifted", "is this dead code", "is this a bug"), dogfood it on the real project corpus and eyeball the finding count and the top hits, not just whether the unit tests pass. Prefer deferring (park on a wip/ branch) over shipping a feature that trains the agent/user to ignore its own output — thousands of false positives is worse than no feature. See also Battle 8's broken-oracle framing: a fixture suite that only tests hand-picked true/false cases is a weaker oracle than a real-corpus run.

Battle 14 — sequential release-watcher deadlock: gate on absolute state, never "changed since I launched"

Rule: This is Battle 6's push-race discipline applied to automation, not just human merge timing. A polling/watcher loop that checks "did state change since I started" is race-prone whenever the state could have already changed before the loop started (a pre-launch publish). Always identify a specific, absolute target to wait FOR (a run ID, a PR merged-state, a version string) — never a relative delta from an arbitrary observation point. Re-measure the pipeline duration periodically (gh run view <id> --json jobs) — CI job mix and runner speed drift over time, and a stale duration assumption is how a watcher's timeout gets tuned wrong.

Battle 15

…(truncated)