Audit Gate Logic

audit-gate-logic — Does the gate actually stop what it claims?

Degree of freedom: MIXED — Phases 0–2.5 [HIGH freedom]; Phase 3 history/probe [LOW freedom — run exactly] (no scratch branch unless asked).

Read-only. audit-cicd asks "is the pipeline fast, cheap, and safe to run." This asks "does the gate stop what it claims to stop." A green check is a claim. The expensive failures are the ones where CI passed and shouldn't have.

Present findings. Do not edit workflows, branch protection, or baselines. Consolidation of the archaeology map → housekeep-gates. Rule-to-mechanical gaps → enhance-agent-guardrails.

This skill vs neighbors

Degrade honestly. Highest-value "is it actually required" checks need branch-protection / rulesets via the repo settings API (gh api), not just YAML. If settings are invisible, audit the workflow files and say so — you can still catch continue-on-error and ratchet classes, but you cannot confirm the required-check list.

How to reason

1. Observe — quote the if:, continue-on-error, required-check list, or history
2. Interpret — can a merge go green while the claimed check did not run or failed?
3. Classify — silent bypass / ratchet game / conflict / archaeology / correct-as-is
4. Severity — ships-broken-green = critical

Worked example

Observe: lint job has continue-on-error: true and is the only required status check named "Lint". Interpret: lint can fail and the required check still reports success. Classify: silent bypass (continue-on-error). Severity: critical — ships-broken-green. Finding: lint | continue-on-error | critical | .github/workflows/ci.yml:24

Phase 0 — Map every gate and its enforcement [HIGH freedom]

Inventory the full gating surface, not just the obvious jobs:

• Workflows & jobs (.github/workflows/*, or GitLab/CircleCI equivalent): each job, its triggers (on:), its if: conditions, path/branch filters.
• Required status checks (branch protection / rulesets): which checks actually block merge vs which merely run.
• Ratchets / baselines: coverage floors, lint-error baselines, bundle budgets, type-error counts, MATCH/DONE burndown, workflow-green-repo green baseline.
• Pre-commit / pre-push hooks: local gates that may not mirror CI.
• Bot gates: CodeRabbit / Sonar / Danger — block vs advise.

For each gate record: what it checks, what happens on failure (block / warn / silently pass), and whether it is enforced server-side or only by convention.

Phase 1 — Silent-bypass failure modes [HIGH freedom]

continue-on-error / soft failure — continue-on-error: true, || true, exit 0 swallowing, or if: always() masking a failure. The check runs, "passes", and enforces nothing.

Required-in-name-only — Runs in CI but is not in the required list → mergeable while failing. Or a required check whose job can be skipped (if:) so it reports success-by-absence. A skipped required check often counts as passed.

Path/branch filters that skip the wrong changes — paths: / paths-ignore: that exclude a gate for the files that needed it (security scan skips .github/, so a PR that disables the scan is not scanned). Filters that never run on the default branch, or only on PRs and not on direct pushes.

Condition conflicts — Two workflows whose if: / triggers both fire on the same event (duplicate work, race on a shared resource, contradictory status posts) or both skip (each assumes the other ran). Missing concurrency groups so two runs race and the loser's failure is discarded.

Fork / permissions bypass — pull_request_target running privileged with untrusted code, or gates that do not run on external-contributor PRs.

Event mismatch — Gate triggers on push but merges use squash/PR (or vice versa), so the enforced commit was never the gated one.

Phase 2 — Ratchet and baseline soundness [HIGH freedom]

One-way that isn't — Can the baseline be regenerated in the same PR that regresses (--update-snapshot, coverage --update-baseline committed alongside the drop), locking the regression in as the new floor? Baseline moves should be a separate, reviewed act.

Exclusion gaming — Coverage/lint passes because the dropped files are in .eslintignore, coveragePathIgnorePatterns, or // istanbul ignore. A ratchet that measures a shrinking surface always passes.

Threshold vs reality — Floor so far below actual that a real drop still clears it (60% gate, 90% actual → 89% "passes"). Global thresholds hiding per-file collapse. Bundle budgets set above current so growth is free.

Baseline staleness / drift — Never regenerated (no longer reflects the codebase) or auto-regenerated every run (never actually ratchets).

Direction bugs — Comparison operator backwards (passes when worse), or comparing against main after main already absorbed the regression.

Burndown ratchet integrity — Can MATCH/DONE mark an item DONE without verification evidence? Can the count go down (work un-done) without failing? Cross-check completion-judge / burndown-full.

Phase 2.5 — Gate archaeology (accreted gates from multiple sessions) [HIGH freedom]

Gates accumulate: each dev, agent session, or "let's add a check" iteration layers another workflow without removing the last. Audit the set:

Duplicate gates, different vintages — Two+ jobs checking the same thing (two lint jobs with disagreeing configs, ESLint and Biome both wired, an old test.yml and a newer ci.yml both running the suite). For each pair: which actually blocks, whether configs disagree, which is vestigial.

Competing baselines / ratchets — Multiple floors for one metric (jest threshold + codecov + a committed floor), updated at different times. The agent satisfies whichever loaded; the strictest one silently stopped being the real gate.

Hook-vs-CI divergence — Pre-commit / pre-push enforcing a different ruleset than CI, so "passes locally" and "passes CI" are different claims and people learn --no-verify.

Dead gates — Triggers that can no longer fire (renamed branches, removed paths, disabled but not deleted). Still cost comprehension: every reader must reason about a gate that does nothing.

Best-of selection — For each duplicate cluster, name the winner:

1. Server-side-enforced beats convention
2. Aggregator-gate pattern (one required check that needs: all others) beats N scattered required checks
3. Merge-queue-compatible (merge_group + matching check names) beats PR-only
4. Auto-tightening ratchet with reviewed resets beats a static threshold
5. AST / structural checks beat regex greps
6. Owned + recently maintained beats abandoned

Record winner + losers per cluster — this is the input housekeep-gates executes.

Phase 3 — Verify against history (where possible) [LOW freedom — do not invent a branch]

Where CI history is inspectable, look for a merged PR that introduced a failure the gate should have caught. Each one is a proven hole, not a hypothetical.

Optional cheap probe on a scratch branch the user already has: introduce a deliberate violation the gate claims to catch and confirm it blocks. A gate never tested against a real violation has unknown efficacy. Do not create that branch or push it unless the user asked.

Definition of Done

• Every gate inventoried with on-failure behavior and server-side vs convention
• All continue-on-error / soft-fail / always() masking found
• Required-check list cross-checked against what actually runs — or explicitly skipped because settings were invisible
• Path / branch / event filters checked for excluding the changes that most need gating
• Cross-workflow both-fire / both-skip / races surfaced; concurrency groups checked
• Fork / pull_request_target / external-PR bypass assessed
• Every ratchet checked for same-PR overwrite, exclusion gaming, slack threshold, staleness, direction bug
• Burndown / completion ratchet integrity checked against its evidence requirement
• History scanned for regressions-shipped-green; deliberate-violation probe noted or skipped with reason
• Duplicate-gate clusters, competing baselines, hook-vs-CI divergence, and dead gates mapped; a winner named per cluster
• Read-only — findings presented, gates not edited; consolidation handed to housekeep-gates

Self-critique before reporting [LOW freedom — do not skip]

1. Evidenced — YAML line, gh api result, or merged PR — not "probably bypassable"
2. Settings honesty — if branch protection was invisible, say so; do not invent the required list
3. Severity justified — critical = ships-broken-green
4. Right owner — consolidation → housekeep-gates; cost → audit-cicd
5. Winner named — every duplicate cluster has a winner + why

Output format

1. Gate inventory — gate | checks what | on-failure (block/warn/silent) | enforced where
2. Bypass findings — gate | failure mode | severity (critical = ships-broken-green) | evidence
3. Ratchet findings — ratchet | weakness | can it lock in / be gamed? | fix
4. Conflict map — workflow A × workflow B | both-fire / both-skip / race
5. Archaeology map — duplicate cluster | members | winner (+ why) | losers to retire
6. Proven holes — merged PR / probe that passed a gate it shouldn't have
7. Fix plan — silent-bypass first, then ratchet integrity, then conflicts. Consolidation → housekeep-gates; pipeline-side → audit-cicd; rule-to-mechanical gaps → enhance-agent-guardrails. Present and stop.