Pre-push verification gate debugging
Repos with heavy local gates block git push until every lane passes (lefthook pre-push hook
calling a verify script that runs docs, unit, E2E, infra and frontend lanes). This skill is for
operating and debugging that gate without burning whole-gate cycles. The methods here
are generic; keep a repo-specific playbook beside the gate script (lane list + commands,
log layout, repro steps) as this skill's reference when one exists.
Detect and understand the gate
lefthook.yml→pre-push→script+runner(+root). Read the script's header comments FIRST: usage, lane names, exit codes, env vars (VERIFY_MODE,VERIFY_SKIP), self-test, and the "reproduce / bypass / logs" failure contract.- A blocked push ends with
🥊 quality gate (Ns)andexit status 1. The hook buffers lane output — never diagnose from the push output alone.
Workflow
- Read the gate's own logs first. The failure block prints:
reproduce: bun <script> <lane>— run exactly this to re-run one lane (minutes, not the whole gate).logs: <dir>/<lane>.log— the lane's full output;<dir>/progress.logis the run timeline (mode, which lanes ran, PASS/FAIL per lane).- Log dir is per-worktree (hash of the worktree path) — find it by mtime (
ls -t), not by recomputing the hash.
- Run one lane for fast iteration:
bun <script> <lane>(e.g.api-e2e). Only re-run the full gate (<script> pre-push) when the lane is green. - The gate tests the WORKING TREE, not the pushed ref. If you push from a different checkout than the one you want verified, the lanes verify the wrong bytes. Gate-test the tree you intend to push.
- E2E/infra lanes:
- They start their own AppHost (Docker: emulator containers) OR reuse a live dev stack that is already ready (probed first). Reuse is convenient but carries state forward.
- Cross-run state contamination: emulator data persists across suite runs on a reused stack. Durable orchestration instance IDs are PERMANENTLY TAKEN after a terminal state, so a later run scheduling the same deterministic ID 409s or silently dedupes. Symptom: a test passes alone, then a DIFFERENT test fails on a later full run with no code change. Fix: fresh ephemeral volumes (kill the reused stack, let the lane start its own).
- Worktree gotchas: a bare
git worktree addhas NO node_modules → tooling lanes fail (tscTS2688 "Cannot find type definition file for 'bun'", bun test resolution errors). Fix: symlink the main checkout's node_modules into the worktree. Never commit node_modules. - Manual repro harness (when the lane's test output hides the real error — e.g. an
orchestration ends
failedwith an EMPTY error field):- Start the infra exactly as the lane does (see the reference / lane source for the AppHost command and env).
- Build the host under test and run it with the E2E settings (the lane copies a
local.settings.e2e.json-style file into the build output before the tests run). - Drive the failing flow with curl, sending the app's auth header (see reference).
- Read the HOST's own logs: orchestration/activity failures appear there with the full
exception ("Task 'X' failed with an unhandled exception: ..."), never in the HTTP status
payload.
failed+ emptyerrorin the API body = the detail is in the host logs.
- Auto-repair: some gates try to regenerate drifted artefacts (baseline.json, index.json) before failing. If repair "declined", the cause is NOT drift — read the lane log.
Gotchas
- A push can LOOK hung while the pre-push hook is simply running.
git push 2>&1 | tailbuffers everything, so the lefthook banner (🥊 lefthook ... hook: pre-push) never shows and the full gate (which can take 5+ minutes) looks like a network stall — zero output, no timeout. Diagnose by re-running the push UNPIPED or in background and reading the first lines: a banner means the gate is working, not hanging. If the user has said "skip gates" / "just push",git push --no-verifyis the sanctioned bypass and returns instantly — do not keep waiting on a gate the user already waived. fullvslimitedmode changes which lanes run; limited skips infra/e2e and prints that coverage is still owed. UnsetVERIFY_MODEfor the real gate.VERIFY_SKIP=<lane>and--no-verifybypass the gate entirely — prefer fixing the lane.- A 401 with "Missing or invalid x-functions-key" can really mean a missing auth header on your curl, not a missing key — check the app's principal/auth scheme before assuming.
- Don't re-run the full gate to test one fix — single-lane runs are minutes faster.
- Force-pushing a rebased PR branch re-triggers remote CI; the local gate still runs against your working tree, so keep the local tree on the branch you are pushing.
- A green full-tree lint does NOT prove the changed-file lint lane passes. If the gate's
lint lane lints only changed files (git-diff scope) while your manual run covered every
tracked file, the lane can flag findings the full run reports clean — e.g. a W0108
unnecessary-lambdaon a freshly addedresolve=lambda: f()that a whole-repopylint $(git ls-files ...)rated 10.00/10. Reproduce with the actual lane (<script> <lane>, e.g.verify.sh pylint) before pushing, not with a full-tree equivalent. - The interpreter you run the gate with determines what the gate's subprocesses can import.
Gates that spawn
sys.executable(scaffold/regeneration lanes) inherit the python that launched the gate. On macOS the barepython3is often the CommandLineTools build (no site-packages, e.g.ModuleNotFoundError: No module named 'jsonschema') — always run lanes with the repo's venv python, or rely on the gate's own_resolve_pythonfallback. - A worktree's own broken
.venvshadows the main checkout's good one. If the gate's python resolution prefers$PWD/.venvand the worktree has a bare/broken venv (e.g. no pip, no deps), every lane fails with import errors even though the main checkout's venv is populated. Fix: delete the worktree.venvso resolution falls back to the main checkout's venv. A stray venv you did not create is the first suspect when a fresh worktree's lanes fail on missing modules. git rebase --continuehangs: the pre-commit hook wedges in rebase context. Lefthook's pre-commit can hang indefinitely while a rebase is being continued (observed with the code-review-graph hook; the git process times out waiting on it and the rebase state survives the kill). Replay the rest of the rebase with hooks off:GIT_EDITOR=true git -c core.hooksPath=/dev/null rebase --continue. This is safe because the pre-push gate re-runs every relevant lane on the pushed tree — enforcement happens at push, not at rebase. A normal commit on a clean branch is unaffected; only the rebase context hangs.- The gate runs SCOPED lanes, not the full list.
progress.logshows e.g.PUSH lanes: docs terraformwhen a branch only touches docs+infra — the gate picks lanes by changed paths. Don't read a scoped run as proof the E2E/dotnet lanes passed, and don't be surprised a scoped run finished in seconds.self-testtooling tests ride inside thedocslane. - Playwright-style e2e lanes reuse a live dev server on a fixed port (
reuseExistingServer: !CI). A wholesale failure pattern — every page test timing out while the [WebServer] log floodsCannot find module/ TS2307 for imports — is a SERVER-level failure, not your change. The reused server may be a zombie from a removed worktree (its node_modules symlink is gone), or the gate's own server hit a transient gap while a concurrentnpm installin the main checkout churned package dirs. Diagnose in this order: (1) who owns the port —lsof -nP -iTCP:<port> -sTCP:LISTEN, thenlsof -p <pid> | grep cwd(never kill a process whose cwd is the owner's checkout); (2) does the failing module exist on disk; (3) re-run the e2e lane alone for fast feedback; (4) re-push. Keep a repo-specific playbook (port, lane commands, worktree symlink quirks) beside the gate when the project has one. - After an auto-repair commit, push AGAIN. The gate's repair path regenerates drifted
artifacts (e.g.
docs/meta/baseline.json,trust-index.json), commitschore(gate): regenerate stale artifacts, and exits non-zero with "push again" — the first push deliberately fails. Re-push the same ref (plain push; after a rebase,--force-with-lease). - A lane can be red because the gate tests a working tree polluted by ANOTHER task's
uncommitted files — the repair path refuses to touch them, so the lane stays red until that
task commits. Repair declines with "not regenerating: has uncommitted changes, and
the repair may only commit bytes it wrote itself". This is not your diff. Prove it before
bypassing: (1) your branch diff contains no file in the failing lane's scope
(
git show --stat HEAD/git diff origin/main...HEAD); (2) the lane's own output shows your files clean. Then the gate's failure block itself names the sanctioned escape:VERIFY_SKIP=<lane> git push— stage-scoped, NOT--no-verify(skips every lane) and NOT fixing the other task's files. The skipped lane still runs on GitHub CI only if CI has it; a docs lane is often local-only, so the PR is unaffected. git push --delete <branch>also runs the pre-push gate and can fail on it; bypass withgit -c core.hooksPath=/dev/null push origin --delete <branch>— deleting a remote ref needs no verification.
References
The gate script's own header + logs are the primary reference (usage, lane names, exit codes, reproduce/bypass commands). Keep a repo-specific playbook beside the gate script with the lane list and commands, log layout, AppHost command, E2E auth header + config, manual repro steps, and any repo-local drift/state files — and update it when the gate changes.