Observability Triage

Observability triage

Query Cloudflare Workers Observability for prod errors, count them correctly, and skip the noise that has already been investigated to a dead end. Apply the known-noise list below without re-investigating those entries.

Access

• Resolve the account at runtime — never hardcode it: the cloudflare-api MCP server pre-binds accountId in mcp__cloudflare-api__execute, and npx wrangler whoami prints it. The workers to triage are the ones this repo deploys (see alchemy.run.ts): the main app worker plus the aux workers (audit engine, landing, self-host).
• Query via the cloudflare-api MCP server (mcp__cloudflare-api__execute). If its tools are absent, run its authenticate flow and give the user the URL — wrangler's OAuth token gets a 403 on the observability API (missing scope), so don't burn time on curl-with-wrangler-token.
• PostHog is the second error source but cannot see exceededMemory / canceled / responseStreamDisconnected outcomes — worker-outcome questions are answerable only here.

Query recipes (verified shapes)

POST /accounts/{account_id}/workers/observability/telemetry/query. All of these are load-bearing; the API's 400s are opaque:

• queryId: "adhoc" is required. Timeframe is epoch milliseconds: timeframe: { from, to }.

• Count invocations by outcome:

  {
    "queryId": "adhoc",
    "timeframe": { "from": 0, "to": 0 },
    "parameters": {
      "datasets": ["cloudflare-workers"],
      "filters": [
        { "key": "$metadata.type", "operation": "eq", "value": "cf-worker-event", "type": "string" }
      ],
      "calculations": [{ "operator": "count", "alias": "count" }],
      "groupBys": [
        { "value": "$workers.scriptName", "type": "string" },
        { "value": "$workers.outcome", "type": "string" }
      ],
      "limit": 100
    },
    "view": "calculations",
    "limit": 100
  }
  

• The cf-worker-event filter is what makes counts mean invocations; without it you count every log line.

• Raw samples: view: "events", empty calculations/groupBys, small limit. Events carry $metadata (service, trigger, level, message, fingerprint, requestId) and $workers (outcome, scriptName, scriptVersion, wallTimeMs, cpuTimeMs, eventType).

• Error-level app logs grouped by message: filter $metadata.level eq error, groupBy $metadata.message.

• Verified filter operations: eq, neq. Percentiles: the API wants "median", not "p50". Filter for substrings client-side on fetched events.

• Useful drill-downs: groupBy $metadata.trigger (route), $workers.scriptVersion (did a deploy change the rate mid-window), $workers.event.request.headers.user-agent.

Counting gotchas

• Grouped results are unsorted and effectively capped (~10 rows returned regardless of limit). A missing group ≠ zero. To see error outcomes, add $workers.outcome neq ok instead of hoping the error rows make the cut; sort client-side.
• One isolate death fans out. An OOM kills every request pinned to the isolate at the same instant — cluster raw OOM events by timestamp before reading the count as user impact.
• Message prefixes fragment groups. Logs with leading timestamps (better-auth's format) split one error into N single-count groups; grouped-by-message counts badly understate them. Sample events and merge client-side.
• Prod deploys are manual — main being fixed doesn't mean prod runs the fix. Check $workers.scriptVersion and the scripts' modified_on before concluding a fix didn't work.

Known noise — filter these out, do not re-investigate

Entries land here only after an investigation proved there is no first-party emit site to fix or demote. Each keeps the one condition that would make it real signal again.

1. SAM chat Durable Object lifecycle (close code 1006)

• Messages (one phenomenon, counted three ways): Connection closed: this Durable Object instance is no longer active. Reconnect or retry the request. (hibernation/eviction), Durable Object reset because its code was updated. (deploy), plus the paired invocation summary whose $metadata.error is close.
• Identify by: eventType: "hibernatableWebSocket", entrypoint SamChatAgent/OnboardingChatAgent, webSocketType: "close", code: 1006, wasClean: false, outcome: "exception", single-digit wallTimeMs, cpuTimeMs: 0, no stack. Fingerprints 3aa4cac26653d09a0a41100a33d413ae (exception), 0ae15457af49b4d9a117eeecf66b040a (summary).
• Why unfixable: the DO is destroyed under the JS — its IoContext is already aborted when workerd delivers webSocketClose, so the first await never settles. partyserver already try/catches the whole close path; an onClose override would catch nothing.
• Nothing breaks: transcripts persist per message in DO SQLite, PartySocket reconnects unconditionally, and credit metering (onChatResponse) never fires on an aborted turn.
• Real signal: a sustained rise that does not correlate with a deploy (would mean mid-conversation evictions beyond hibernation).

2. Network connection lost.

• Identify by: fingerprint be89d4ff7a64cb4d4dceae0f51cfe708, or the message verbatim. Runtime-generated shape: source.level absent, source.exception present, $metadata.origin: "fetch" — the opposite of every app log (source.level present, no exception).
• Why unfixable: workerd's own record of a client disconnecting mid-stream; the exception never enters app code — the sibling request event for the same requestId has outcome: "ok". No emit site exists; wrangler.jsonc observability config has no per-message filter. Do not add try/catch around the stream handlers (dead ceremony).
• Real signal: if the /mcp share of this group grows, treat it as a tool-call-latency symptom (clients timing out and cancelling) and route it to the /mcp performance track — not to this log group.

Runtime-vs-app litmus test

Before investigating any unfamiliar error event: source.level absent + source.exception present ⇒ the Workers runtime wrote it, not the app. There is no call site to grep for; judge it by the sibling request's outcome.

Adding an entry

Add to this list only after establishing there is no first-party emit site (grep the message; check the runtime-vs-app litmus above) and nothing is left in a wrong state. Every entry must include identify-by markers (fingerprint if stable) and its real-signal condition.