Debug

/debug - Debugging Quick Reference

Fast reference for known production failure patterns. For deep debugging, use /carmack.

Mandatory upstream-protocol check (third-party integrations)

When the symptom involves a third-party integration falling back to a worse path (handoff, redirect, "manual step required") — Verint dform, Clerk, Stripe, SeeClickFix, OAuth, any SaaS — load ~/.claude/skills/shared/upstream-protocol-investigation.md BEFORE proposing a fix. Read the upstream's primary client source, capture real network traffic, inspect rendered data-* attributes. Treat any "Verified YYYY-MM-DD" comment in our codebase as a hypothesis to re-verify, not a fact. Token cost is unlimited for this; user explicitly authorized it (2026-05-09 SF311 graffiti incident: bandaid 0b746a4 → real fix f27d1e3 after reading dform's api.js lines 462–520).

DocuSeal template-vs-submission gate

When a DocuSeal form is missing fixed business values, load ~/.claude/skills/shared/docuseal-template-contract.md. Prove whether the signer used a unique submission URL or the template shared link; inspect both the live submission and live template because mocked payload tests cannot detect template default drift. For multi-party templates, a public shared link is unsafe unless party ownership is explicit; AIVA's HIPAA template must keep it disabled. /debug remains read-only for production state.

Mutating symptom across fixes = wrong premise (MANDATORY — 2026-05-19)

When each fix makes the symptom change instead of disappear — fetch_error → handshake_rejected → publicid_timeout, every "fix" peeling back to reveal a new failure one layer deeper — STOP. That staircase is the tell that the whole feature premise is wrong, not that you are three fixes from done. Before fixing layer N+1, verify the premise itself: does the real client even use this code path? Capture live traffic and confirm. Reference incident: the SF311 cable chain — four green-tested fixes (wss://→https://, missing Origin header, subscription identifier, publicId timeout) each surfaced the next layer; a live mitm capture then proved the real app never opens the WebSocket at all — the whole path was dead code. Many round-trips of "fix the next layer" should have been one round-trip of "verify the premise."

Recurring symptom after a DOCUMENTED remediation = the runbook's causal model is wrong (MANDATORY — 2026-08-03)

The third sibling of the two rules above. Mutating error = wrong premise. Static error = missing discriminator (#32). A symptom that keeps COMING BACK on a schedule, after you keep applying the documented fix, = the runbook is wrong about what causes it.

The trap is that the remediation appears to work every time. It restores service, so it never reads as a failed fix — it reads as an unlucky recurrence. That is what lets a wrong causal model survive N repetitions. Executing a runbook is not debugging, and "the alert told me to" is not a diagnosis.

Tripwire — fires on the SECOND repetition, not the fifth: if a documented remediation has been applied ≥2 times and the symptom returned, stop executing it and ask the two questions that the runbook is silently answering for you:

1. What actually MINTS the thing I keep replacing, and does my system ever call it? Replacing an artifact is not the same as participating in its lifecycle. If nothing in your code ever calls the issuer, you are refilling a bucket with a hole and the interval between refills is the hole's size.
2. Measure the true survival time, don't inherit the folklore number. Time the interval yourself; the remembered figure is usually the longest one anyone noticed, and the real number is a much sharper clue.

Reference incident (2026-08-03, AIVA Google Voice). An alert said "session expired — reseed the cookies," with a copy-paste command. It was followed three times in 18 hours. Each reseed restored SMS, so each expiry read as bad luck. The runbook was wrong: __Secure-1PSIDTS/__Secure-3PSIDTS are minted ONLY by accounts.google.com/RotateCookies on a 600s cadence and gate every RPC, while the Worker talked only to clients6.google.com, which never returns them. Nothing in the system ever called the issuer, so reseeding could never have fixed it — it just restarted the same clock. The cookie-refresh code was present and correct; it simply had nothing to persist. Question 2 mattered too: the folklore said "about an hour," but timing it gave **20 minutes** (21:51 ok → 22:11 alerted), which matches the 600s rotation cadence and pointed straight at the mechanism. Fix = call the issuer on the cadence it declares.

Corollary — "SAPISID present=true" class of false-healthy signal. The alert reported the long-lived credential as present while the short-lived one that actually gates access had already expired. When a health line reports a credential/resource as OK during a confirmed outage, check whether it is reporting a different-lifetime component than the one that fails. Enumerate the credential set by lifetime before trusting any "present=true".

PREMISE-CHECK GATE — run BEFORE the first fix (not just on the staircase). Full rule: ~/.claude/skills/shared/premise-check.md. Two questions, answered against LIVE upstream docs (not a cached /skill note): (1) Is this approach even valid for THIS runtime/SDK/platform? The browser SDK ≠ native SDK ≠ server SDK — a strategy/option/API documented under one is routinely absent or forbidden in another. If every doc/example for the thing you want sits under a different platform than yours, that's your answer — stop. (2) What's the cheapest probe (curl the API / grep the installed bundle / read the doc's "supported platforms" line) that proves it's possible here? Run it before coding. Docs-before-note (always): a /skill recipe, comment, memory, or prior conclusion is a HYPOTHESIS — re-verify any load-bearing "API/SDK can/can't do X" claim against the upstream's own current docs/source before building on it; live source wins, fix the stale note in the same pass. 2026-06-13 reference incident: hours spent debugging Clerk native oauth_token_apple from a Capacitor webview — clerk-js (browser SDK) can never send it (browser-forced Origin vs Clerk Native-API Authorization conflict), a fact Clerk's docs state plainly under "Expo only." A wrong /ios trap-#4 note was trusted as fact; the working web-OAuth fix was a 5-minute live-doc read away. Now also enforced session-wide by the premise-check-session-start.sh SessionStart hook.

"Every subset passes" / "it looks intermittent" = your manipulation never applied (MANDATORY — 2026-08-25)

The fourth sibling of the three rules above. Mutating error = wrong premise. Static error = missing discriminator (#32). Recurring-after-remediation = wrong runbook. Every arm passing, no single change reproducing it, and a growing urge to call the bug "intermittent" = the change you think you applied never reached the system, and you are A/B-ing A against A.

A deterministic bug measured through a no-op'd manipulation presents exactly as flakiness, because you are re-sampling one unchanged condition. So "it's intermittent" is a claim requiring evidence, not a fallback explanation.

Tripwire — before writing the word "intermittent", and before any bisect round 2: name an observable that MUST differ between arms, read it back after each apply, and abort the run if the arms look identical. Not "the apply exited 0" — an observable in the running system.

Reference incident (2026-08-25, SmartTube 403). Bisected 21 profile files to find which setting 403'd every video stream. Every subset passed; the full set failed once then passed twice; I concluded "the PoToken mint is intermittent, not a config bug" and posted it publicly. It was one deterministic line (preferred_dns_type=2 → Google DoH resolver). The app's own Auto backup rewrote the staged fixture on every launch, so each "restore" was a silent no-op and every run tested a fresh profile — and I had suppressed the restore step's output, so nothing surfaced it. One post-restore read of the sidebar order (custom vs default) would have caught it on run 1. With the fixture-rewriter disabled and each restore verified, the bisect converged in three rounds.

Full gate + instrument-trap table: ~/.claude/skills/shared/experiment-manipulation-check.md.

Route-coverage bugs: get the site's route list before you sample (2026-07-31)

When the symptom is about which pages/endpoints exist or work — "page X 404s in prod", "some routes broke after the deploy", "the new pages aren't live", "search can't find our docs" — do NOT debug from the handful of URLs you happened to click. Get the declared inventory first, then diff it against reality:

curl -sL https://<host>/robots.txt | grep -i sitemap
curl -sL https://<host>/sitemap.xml | grep -o '<loc>[^<]*' | sed 's|<loc>||' > /tmp/declared.txt
# now diff DECLARED against DEPLOYED — the bug is usually in the gap
while read -r u; do printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' -m 10 "$u")" "$u"; done < /tmp/declared.txt | grep -v '^200'

For an API, the analog is the published spec (/openapi.json, /swagger.json, /api-docs, GraphQL introspection, or unbrowse_eval_spec_discover) — not a sitemap.

This is the "Compared to What?" rule applied to debugging: a sampled set of working URLs is a delta, the declared route list is the denominator. Two failures it prevents: declaring "routes are fine" after checking the three you remembered, and declaring "page X doesn't exist" when it exists but 404s (a very different bug).

Distinct from the security-baseline check below, where a missing sitemap.xml is itself the finding (#16). Here the sitemap is the instrument, not the defect.

No-Lie Post-Fix Gate (MANDATORY — 2026-05-18)

A "fixed" symptom is one that fails to reproduce on the deployed/built artifact, not in the source. After EVERY fix, before declaring root cause resolved, run the No-Lie Verification Protocol from ~/.claude/skills/shared/no-lie-verification.md Checks 3 + 4:

1. Symptom re-test on the artifact, not the source. Examples:
   • "Stale copy" bug → cache-busted curl https://prod-url/path for the OLD string. Expected: 0 matches.
   • "API 500 on category X" bug → curl -X POST reproducing the original request with a fresh cache-buster. Expected: 200 or the deliberate fallback, never a regression.
   • "Race condition in worker" bug → run the original repro harness (tools/repro/*.sh) under load and confirm the failure no longer reproduces over N runs (N≥10).
2. Whole-repo symptom re-grep, not just touched files. Proves "fixed all instances", not "fixed the ones I noticed". The whole-repo grep must return 0 matches OR only matches that are demonstrably unrelated (JSON values, log files, third-party node_modules).
3. If the fix is in TRIGGERED behavior (failover, retry, fallback, circuit-breaker, rate-limit cooldown, error/catch branch, conditional cron) — induce the trigger and watch the path fire (Check 6 in no-lie-verification.md). Don't infer it from "the parts work": force the 429 / fail the dependency N times / feed the exact bad input on an isolated copy, confirm the fallback/retry/error-path actually executed, then confirm the live instance is untouched. "Configured" ≠ "fires."
4. Every claim in the final report cites the command that proves it. Forbidden without proof: "fixed", "verified", "root cause confirmed", any specific count.

Reference incident (2026-05-18): /carmack agent reported "no stale '10,000' strings — rg clean" but never curl'd the live URL. Old build was still serving until /ship deployed. The fix wasn't a fix until the deployed artifact was re-tested.

Usage

/debug [pattern name or symptom]

Examples

• /debug catch-all -- Catch-all error handling masking root cause
• /debug react undefined -- React "X is not defined" scope bug
• /debug silent startup -- React silently fails to mount
• /debug auth failed -- Generic auth error hiding real cause
• /debug broken icons -- Third-party icons/images missing (CSP blocking)
• /debug text overflow -- Text escaping card boundaries on mobile
• /debug stale data -- Admin changes not visible to users
• /debug deploy logout -- Users logged out after every deploy
• /debug cloudflare security alert -- Cloudflare flagged site for missing security.txt / HSTS / CAA / DNSSEC
• /debug slow page -- Page/route slow: full-table scan on the hot path, cron warming the wrong cache, or a lagged-source window returning empty
• /debug cookies -- Need cookies for curl/yt-dlp/scrape from a logged-in browser session (see "Cookie extraction" below)

Pattern numbering (read before adding one)

Pattern numbers are globally unique across all reference files, not per-file. Before adding ## Pattern N:, claim the next free N:

grep -rhoE "^## Pattern [0-9]+:" ~/.claude/skills/debug/references/*.md \
  | grep -oE "[0-9]+" | sort -n | tail -1        # highest in use
grep -rhoE "^## Pattern [0-9]+:" ~/.claude/skills/debug/references/*.md \
  | grep -oE "[0-9]+" | sort -n | uniq -d        # existing collisions

#12 and #15 are legacy collisions (two distinct patterns each). Always cite a pattern as file + number (error-handling-patterns.md #29), never a bare number, so a collision can never misroute an agent.

Pattern Routing

Match the user's symptom to the right reference file, then load ONLY that file.

| silent startup, blank page, no console errors, module-level throw | Silent React Startup (#3) | react-patterns.md | | useEffect, renders twice, state lags, derived state | useEffect Abuse (#15) | react-patterns.md | | localStorage, preference lost, resets on refresh | Preference Lost on Reload (#13) | react-patterns.md | | double click, duplicate API call, async button | Async Button Double-Click | react-patterns.md | | chat/support widget gone after SSR, cookie banner missing on SSR page, ?support=open/deep-link does nothing, global widget vanished after Hono/Astro/RSC conversion, "worked on every page before SSR", floating launcher dead, analytics/exit-intent dropped on SSR route | Global App.tsx Component Vanishes After SPA→SSR Conversion (#24) | react-patterns.md | | invisible text after SSR, white headings on light/gray bg, page background wrong color after conversion, "colors got messed up on conversion", body background overridden, SSR design clobbered by Tailwind/island CSS, text disappeared but HTML is there, computed bg ≠ design token, low contrast only on SSR pages, island/global CSS leaks body styles | Bundled Island/Global CSS Clobbers SSR Inline Design — Invisible Text (#25) | react-patterns.md | | auth guard, unauthenticated error, no redirect | Missing Frontend Auth Guard | react-patterns.md | | renders undefined, shows NaN, "Invalid Date", "[object Object]", toggle/section/control disappeared or silently vanished, "make sure nothing is undefined", Cannot read properties of undefined, is not iterable, null-gate hides UI, useState<T\|null> gates a render, .map/property access on possibly-undefined API data, admin/dashboard renders garbage, optional chaining missing | Undefined / Null-Render Safety (9-pattern catalog + live DOM grep for undefined/NaN/[object Object]) — also re-sweeps the adjacent admin blind-spot class (default-LIMIT truncation, NULL-aggregate sort burial, count-source mismatch, inner-JOIN row drop, admin-auth DB fallback) | ~/.claude/skills/shared/undefined-null-render-safety.md | | generic isRecord/isObject guard, as unknown as T, (x as any).field, repetitive type-guard boilerplate, "vibe coding" / AI-slop TypeScript, loose unknown-everywhere, no schema validation at boundary, Record<string, unknown> everywhere, input as object as User, missing // SAFETY: on a cast, widen-then-assert, vi.mock module mocking, "anti-slop findings" / oxlint anti-slop/* errors | Anti-Slop TypeScript — replace generic guards/casts with named types, discriminated unions, or Zod (z.infer). Enforcers: repo-vendored dmmulroy/anti-slop Oxlint plugin (./node_modules/.bin/oxlint when tools/oxlint/anti-slop/ exists; vendor via the /install-anti-slop skill) or fallback detector ~/.claude/skills/carmack/tools/detect-ts-slop.sh. AUTO-FIX LOOP UNTIL 0: fix every finding in source by adding evidence (inference, satisfies, boundary parsing, genuinely-checked // SAFETY: invariant), re-run enforcer + tsc --noEmit, repeat to 0; a finding surviving 5 attempts → surface to the user. Never fix by weakening rule severity, oxlint-disable, or laundering types | ~/.claude/skills/shared/anti-slop-typescript.md | | text overflow, min-w-0, flex escape, card boundary | Text Overflow Flex+Grid (#12) | css-layout-patterns.md | | grid mobile, grid-cols, responsive breakpoint | Fixed Grid Breaks Mobile | css-layout-patterns.md | | iOS Safari, blank iPhone, PDF iframe, vh units | iOS Safari Rendering | css-layout-patterns.md | | og image, twitter card, social cache, stale card | Social Card Cache (#5) | csp-cache-patterns.md | | CSP, embed blank, frame-src, img-src, broken icons | CSP Blocking (#9) | csp-cache-patterns.md | | YouTube "Error 153", "Video player configuration error", embed shows gray panel / "Watch video on YouTube", Vimeo/Spotify/SoundCloud embed won't load, video plays on youtube.com but not on my site, embed broke after adding security headers / secureHeaders() / Hono | Embed Dies With No Referer (#27) — NOT CSP, NOT the video. The page sends Referrer-Policy: no-referrer (Hono secureHeaders() default; also .htaccess / WP security plugins / ad blockers). YouTube has refused Referer-less embeds since late 2025. Probe FIRST, 30s: curl -sI https://<site>/ \| grep -i referrer-policy (→ no-referrer = confirmed) and curl -s -o /dev/null -w "%{http_code}" "https://www.youtube.com/oembed?url=https://www.youtube.com/watch?v=<ID>&format=json" (200 = video is fine; 401 = embed disabled; 404 = dead). Fix = referrerpolicy="strict-origin-when-cross-origin" on the iframe — never weaken the global header. Verify on the DEPLOYED artifact (hono/jsx vs React attribute casing can drop it) + a real browser screenshot; allow >30s for Worker propagation before concluding. Sweep the repo for the sibling bugs: rendered hrefs containing whitespace (renderer swallowed prose into the URL) and bare URLs rendering as dead text. | csp-cache-patterns.md (Pattern 27) | | Cloudflare security alert, Security Insights CSV, security.txt missing, /.well-known/security.txt 404, securityheaders.com low score, observatory grade D, missing HSTS/CSP/COOP/CORP, CAA records, DNSSEC, RFC 9116, sitemap.xml 404, robots.txt advertises sitemap that doesn't exist | Site Security Baseline (#16) | ~/.claude/skills/shared/site-security-defaults.md | | CF Security Insights flags: "Dangling A/AAAA", "Unproxied CNAME", "DMARC Record Error", "Bot Fight Mode not enabled" — DO NOT blindly apply CF's recommended fix; 16% are false positives that break Clerk/SaaS CNAMEs, delete live Google-forwarding DNS, or duplicate valid DMARC. dig/curl-verify the OFTEN-FALSE classes first. Apply only the 3 safe fixes (security.txt + block AI bots + AI Labyrinth). | CF Security Insights triage map | `/.claude/skills/shared/site-security-defaults.md(triage section) +/.claude/skills/carmack/tools/cf-security-insights.sh| | **DMARC failing from a sender IP, custom-domain mail bouncing / landing in spam, "send-as alias DMARC fail", Gmail "Send mail as",p=rejectrejecting my OWN mail, "SPF passes but DMARC fails", "can't send from my domain", forwarded mail failing DMARC, a DMARC aggregate (RUA) report flags a source** — this is OUTBOUND auth alignment, NOT broken records. Records are usually fine; the *sending path* is wrong (Gmail send-as / a relay signs as the wrong domain → no aligned identifier → DMARC fail).dig the real SPF/DMARC/DKIM/MX FIRST; adding the relay's IP to SPF usually does NOTHING (SPF aligns on the envelope domain). **Check the platform's CURRENT native send capability before recommending a 3rd-party — verify live, not from memory**: a Cloudflare domain can now SEND via Email Sending (wrangler email sending list/settings; smtp.mx.cloudflare.net:465, user api_token, pwd = CF token w/ Email Sending Write). Fix = route the From-domain's sending through a DKIM-aligned sender; verify with a REAL send (gog send --from → grepdmarc=pass). | Email Deliverability / DMARC Alignment | /.claude/skills/shared/email-deliverability-dmarc.md| | **provider swap, replace Resend/Auth0/Stripe/S3/Twilio with X, migrate email/auth/payments provider, "switch from X to Y", removed the old SDK, deleted the old API key, feature silently stopped after swapping providers,if (!env.OLD_KEY)gate, legacy ids unresolvable, old provider ids in DB** | **Provider Migration Safety** — four silent breakages, none of which throw: (1) feature gates still testing the OLD provider's env var (incl.CRITICAL_ENV_VARS) → feature disabled forever once the secret is deleted; (2) persisted legacy identifiers the new API can't resolve → zero rows → a confident wrong status; (3) the new SDK returns {data,error} instead of throwing → every failure reads as success; (4) the local emulator (wrangler dev) doesn't enforce the new provider's server-side rules → local green ≠ prod accepted. Run the 7-item checklist. **Never decommission the old account off one repo's grep** — another project may hold a live key on the same verified domain. | /.claude/skills/shared/provider-migration-safety.md| | **status chip says "expired"/"not found" on rows that are fine, freshly-created record reads as missing, a uniform block of rows all show the same scary status, status derived from analytics/GraphQL/search backend, zero rows treated as a reason,retention_expiredon brand-new data** | **Absent Data Reported As A Confident Wrong State (#29)** —rows.length === 0has ≥4 causes (ingestion lag, wrong id namespace/provider, real retention expiry, never created, query failed) and they need different words. Discriminate on identifier SHAPE before querying (free), use the system of record'screated_atvs the backend's ingestion grace window to separate lag from expiry, model reasons as a union, and never render a raw enum in the UI. |error-handling-patterns.md(Pattern 29) | | **third-party submit "worked" (200 + id, no error) but the ticket/record never went live — never got a public number, never opened, never associated,openedAt: null, publicId: null, submittedTickets: 0; guest/anonymous create dead-ends silently; A/B-tested a flag/category/field and the symptom stayed IDENTICAL; "why does spotmobile/guest submit never get a case number"; freshly-minted guest token; shadow-limited actor; "is it the payload or something else"** | **False-Positive Success — async lifecycle is the real signal + IDENTITY is the hidden variable (#30)** — a synchronous 200+id is a *pending* state, not *done*; gate success on the lifecycle transition (public number / opened/ row in public dataset), polled, against a known-good baseline. When fix-A/fix-B/fix-C leave the symptom byte-identical, STOP editing the request — the actor/identity/account you held constant is the variable; re-run the SAME payload under a known-good established identity FIRST. Prove the working backend by RUNNING THE REAL FUNCTION LIVE (repro harness → real id), never from old code or a "Verified" comment. **Fastest confirmer that it's the ACCOUNT, not your code: does the vendor's OWN first-party client (their official app/site) show the SAME stuck symptom under the same identity?** If yes, it's an account/identity-level block (shadow-ban/write-limit) — stop debugging payload/version/token and switch identity or backend. Reference (2026-08-12): Solve SF/submit403'd for improvebayarea AND the official iOS app stalled on "pending" for the same account → account write-block; fix was to abandon the authenticated backend for the account-less Verint public form (nothing to shadow-ban). |error-handling-patterns.md(Pattern 30) | | **async id never resolves / report stuck at "getting case number" forever; a resolver/reconciler/poller cannot find the record OUR OWN system created; matching by category/type/service_name/status returns nothing while the user says the integration "constantly fails"; "did the city/vendor actually receive it?"; authority relabeled our category; outcome counters read all-ok while users see failure** | **The authority relabels your artifact — match on YOUR echoed content, never on labels the authority owns (#39)** — investigation order is MANDATORY: (1) query the AUTHORITY'S OWN record store first (Open311/Socrata/vendor API) around the submit time+place and look for OUR OWN echoed boilerplate in the record body — two curls settle "did it actually happen" before any code is read; (2) fix the matcher to rank fingerprint evidence (our echoed text) above label agreement, and pin the fingerprint to the constant that generates it with a test; (3) THEN audit our measurement path (pending recorded as ok, success_rate defaulting to 100% when unmeasured, give-up branches that can never fire). Instrument traps that fake "the record does not exist": Socrata datasets use LOCAL time while Open311 uses UTC (a 7h miss returns confident empty sets — positive-control every zero); list endpoints silently truncate (Open311 default page_size 50). Reference: 2026-08-26 improvebayarea — 12/12 "stranded" SF refs were REAL filed cases (one already closed by the city) under relabeled service names; a day-old memory asserting "submissions never file" was the tz-blind instrument talking. |error-handling-patterns.md(Pattern 39) | | **a route family shows a high 5xx/error rate but every manual probe returns 200; 522 / 524 /originResponseStatus: 0; "origin unreachable" on a Workers-only zone; errors cluster at :00/:02/:30; empty User-Agent; single-colo errors; error count ≈ a fleet size (cities/tenants/shards); "is this users or us?"; an outage nobody can reproduce; a cron/prewarm/warmer/poller/self-health-check is in the picture** | **Your own cron is the caller — ATTRIBUTE before you diagnose (#40).** The first question is **WHO** is making the requests, not why they fail — one httpRequestsAdaptiveGroupsquery grouped bydatetimeMinute+ UA + colo +cacheStatuseither kills the entire user-facing hypothesis or confirms it, *before* the handler is opened. Self-inflicted signature: clustering at the cron boundary + **empty UA** (Worker subrequests carry none) + one colo +cacheStatus: bypass+ count ≈ fleet size. Platform fact: **a Worker cannotfetch() its own Custom Domain** — CF forwards the same-zone subrequest to the *zone origin*, and a Workers-only zone has none (AAAA 100::, RFC 6666 discard) ⇒ 522/origin=0 every time (workers/configuration/routing/routes; workerd#787); probe with curl --resolve host:443:[100::]→ exit 28. Then check the two siblings: a test that **asserts the buggy behavior** (a mockedfetch returns 200 and *cannot observe* a platform refusal — it pins the bug and stays green for months), and a warmer that **warms nothing** (cf-cache-statusstillMISSan hour after the run). Beware "intermittent" — that is a causal claim needing #38's evidence; here the failures were perfectly periodic and the probes passed only because they never coincided with the cron. Reference: 2026-08-29 improvebayarea — 523/523 of the zone's 522s at :02–:03, ~44/run = 43 cities +/map/oakland, 32.5% measured 5xx, zero users affected. | error-handling-patterns.md (Pattern 40) | | **ibaCLI / improvebayarea.com 311 filing** — "did my 311 ticket actually file", submit returned 200 but no case number,mobile311.sfgov.org/services/case/404s, ticket missing from the recent feed, "the CLI only lists 20 categories", no steam-clean/power-wash category, wrong category filed,recategorized_from in the response | **iba traps (verified 2026-08-01) — three of the four "it failed" signals are FALSE.** (1) **--mode prefillis the DEFAULT and it FILES** — all three modes file; only--dry-rundoesn't. (2) **ref-vs-caseid:** Verint assigns a public caseid synchronously for *some* forms only.pw_street_cleaning→ real101004…; pw_graffiti→valid:true+ a UUID **ref**,caseid_pending:true, and a lookup_caseid_url(improvebayareasrc/sf311.ts:1768-1774, src/index.ts:5142). Resolve with iba lookup (takes a **ref**, NOT a caseid — passing a caseid returnsref_not_found) or iba submit --wait . (3) **False failure signals:** mobile311.sfgov.org/services/case/ **404s by design** for ref-format ids (src/sf311.ts:1770) and even 404s for real filed caseids; /api/recentis a **narrow 25-item sample** (had zero Graffiti-category rows) so absence ≠ failure — use?q=; DataSF vw6y-z8j6lags **1-2 days** and is the only real confirmation. (4) **Capture the FULL submit response** — the id is unrecoverable afterward;ibanow prints a SUBMIT RESULT banner on **stderr** so| headcan't destroy it. (5) **Registry drift:** the baked category map silently ran 20-of-42 — runiba categories --check(diffs vs the live API, exit 1 on drift) before trusting it;--live [--city oakland]for the authoritative list. No steam-clean/power-wash category exists — do NOT force it intomissed_street_cleaning(920010 = "the sweeper skipped my route"). |/.claude/projects/-Users-/memory/reference_local_tools.md+error-handling-patterns.md(Pattern 30) | | **"the deploy broke prod",Failed to load module script ... MIME type "text/html", SPA won't mount after deploy, only SEO fallback text renders, ERR_BLOCKED_BY_CLIENT, page worked 5 min ago, verifying a deploy in a clone browser** | **Your Verification Browser Is Stale — False "Prod Is Broken" (#28)** — a cached HTML shell references pre-deploy asset hashes; the SPA fallback serves index.htmlfor the missing.js; the browser refuses HTML as a module script. **Production is fine; only your tab is broken.** NEVER conclude a bad deploy from a browser alone: curl the live HTML for its asset refs, confirm each returns 200 text/javascript(nottext/html), and grep the deployed chunk for a string only your new code has. location.reload(true)is a no-op; usenavigate_page {ignoreCache:true}or a cache-busting query param.ERR_BLOCKED_BY_CLIENT= ad blocker, not your code. |csp-cache-patterns.md (Pattern 28) | | **npm install` blocked or behaving oddly**, "Socket npm exiting

…(truncated)