Debugging Durable Objects

Debugs Savecraft Durable Objects (SourceHub, UserHub) using the admin introspection API. Use when investigating source connectivity issues, events not reaching UI, DO state problems, daemon connection failures, stale sources, or internal errors. Triggers on "source not online", "events missing", "DO state", "debug source", "debug user", "admin API", "ring buffer", "internal error", or any DO troubleshooting.

joshsymonds a794921 6.0 KB Updated Sep 17, 2026

Integrations & APIs

Install

Press C to copy

$ npx skillmds@latest add joshsymonds/debugging-durable-objects

Installs into ~/.claude/skills (or .claude/skills inside a project). Restart Claude Code to pick it up.

Download SKILL.md

＋ Add to collection

README badge

[![SkillMD](https://skillmd.com/badge/joshsymonds/debugging-durable-objects.svg)](https://skillmd.com/skills/joshsymonds/debugging-durable-objects)

Trust & safety

Audits pending

Not yet audited · commit a794921

Works with: Claude Code Claude.ai OpenAI Codex +

File contents

Debugging Durable Objects

Use the admin introspection API to diagnose SourceHub and UserHub issues. Full architecture details in docs/worker.md under "Debug Introspection".

Auth

All admin endpoints require Authorization: Bearer $ADMIN_API_KEY.

Environment	Where the key lives
Production	`wrangler secret` (set via `wrangler secret put ADMIN_API_KEY --env production`)
Staging	`wrangler secret` (set via `wrangler secret put ADMIN_API_KEY --env staging`)
Local dev	`worker/.dev.vars` → `ADMIN_API_KEY=...`
Tests	`worker/vitest.config.ts` miniflare bindings

Base URLs: production https://api.savecraft.gg, staging https://staging-api.savecraft.gg, local http://localhost:8787.

Endpoints

Discovery (D1)

Endpoint	Returns
`GET /admin/sources`	All sources: `source_uuid`, `user_uuid`, `hostname`, `source_kind`, timestamps. Capped at 200.
`GET /admin/source/:uuid/events?limit=N`	D1 `source_events` rows (default 50, max 500). Includes `event_type`, `event_data`, `created_at`.

SourceHub Debug (per-source DO)

Endpoint	Returns
`GET /admin/source/:uuid/debug/state`	`sourceState` (sources array with online/lastSeen/games), `sourceUuid`, `userUuid`, `sourceMeta`, `alarm` timestamp
`GET /admin/source/:uuid/debug/connections`	`daemonCount`, `connections` array with tags per socket
`GET /admin/source/:uuid/debug/log?level=X&limit=N`	Ring buffer entries (newest first). Levels: `debug`, `info`, `warn`, `error`. Limit capped at 200.
`GET /admin/source/:uuid/debug/storage`	All DO transactional storage key names

UserHub Debug (per-user DO)

Endpoint	Returns
`GET /admin/user/:uuid/debug/state`	`mergedState` (all sources merged), `userUuid`
`GET /admin/user/:uuid/debug/connections`	`uiCount` (active UI WebSocket count)
`GET /admin/user/:uuid/debug/log?level=X&limit=N`	Ring buffer entries. Same filters as SourceHub.
`GET /admin/user/:uuid/debug/storage`	All DO storage key names

Two Data Sources

Source	What it captures	Lifetime	Use for
Ring buffer (`/debug/log`)	All DO activity: connections, mutations, forwarding, errors	In-memory, 200 entries, lost on DO eviction/deploy	Live debugging, recent activity
D1 events (`/events`)	Protocol events + `internalError` catch-block failures	Persistent, last 100 per source	Post-mortem, historical patterns

Ring buffer entries also emit to console.log as structured JSON — use wrangler tail --env production --format json for real-time streaming.

Debugging Workflows

Source not showing online

1. GET /admin/sources → find source_uuid
2. GET /admin/source/:uuid/debug/connections
   → daemonCount: 0? Daemon WS never connected or was dropped.
     Check daemon logs for connection errors.
   → daemonCount: 1+? DO thinks daemon is connected. Continue.
3. GET /admin/source/:uuid/debug/state
   → sources[].online: false? State mutation failed.
     Check /debug/log?level=error for "state mutation failed".
   → userUuid: null? SourceHub doesn't know which user owns this source.
     Source may not be linked. Check D1: source has user_uuid?
4. GET /admin/user/:uuid/debug/state
   → Source missing from mergedState? SourceHub→UserHub forwarding broken.
     Check SourceHub /debug/log for "forward state to UserHub failed".
5. GET /admin/user/:uuid/debug/connections
   → uiCount: 0? No browser connected. UI issue, not DO issue.

Events not reaching UI

1. GET /admin/source/:uuid/debug/log
   → Look for "message received" entries — daemon IS sending events.
   → Look for "forwarding event to UI" — SourceHub IS forwarding to UserHub.
2. GET /admin/source/:uuid/debug/state → check userUuid is set
3. GET /admin/user/:uuid/debug/log
   → Look for "forwarding event to UI" with uiCount > 0.
   → uiCount: 0 means no UI client to receive.
4. GET /admin/source/:uuid/events
   → Events in D1? Then persistence works but forwarding doesn't.
   → No events? Check SKIP_PERSIST set — some event types are intentionally skipped.

Internal errors / silent failures

1. GET /admin/source/:uuid/debug/log?level=error
   → Shows all catch-block errors with context
2. GET /admin/source/:uuid/events?limit=50
   → Filter for event_type = "internalError"
   → Each has: context (which method), error message, stack trace
3. Common internal errors:
   - "forward state to UserHub failed" → UserHub DO unreachable
   - "event persistence failed" → D1 write error (NOT retried to avoid recursion)
   - "config push failed" → D1 source_configs query or WS send failed
   - "state mutation failed" → DO storage read/write error

Stale source (shows online but daemon is gone)

1. GET /admin/source/:uuid/debug/state
   → Check sources[].lastSeen — how old?
   → Check alarm — is the reaper alarm scheduled?
2. The alarm fires every 30s (ALARM_INTERVAL_MS).
   Sources with lastSeen > 90s (STALE_THRESHOLD_MS) get evicted.
3. If alarm is null and source shows online → alarm was deleted prematurely.
   Check /debug/log for "alarm fired" / "alarm rescheduled" entries.

Security

API key comparison uses crypto.subtle.timingSafeEqual (not ===)
Debug subpaths are allowlisted: only state, connections, log, storage
Admin endpoints are read-only — no state mutation possible through them
The admin router is in worker/src/admin.ts

---name: debugging-durable-objectsdescription: Debugs Savecraft Durable Objects (SourceHub, UserHub) using the admin introspection API. Use when investigating source connectivity issues, events not reaching UI, DO state problems, daemon connection failures, stale sources, or internal errors. Triggers on "source not online", "events missing", "DO state", "debug source", "debug user", "admin API", "ring buffer", "internal error", or any DO troubleshooting.---# Debugging Durable ObjectsUse the admin introspection API to diagnose SourceHub and UserHub issues. Full architecture details in `docs/worker.md` under "Debug Introspection".## AuthAll admin endpoints require `Authorization: Bearer $ADMIN_API_KEY`.| Environment | Where the key lives ||-------------|-------------------|| Production | `wrangler secret` (set via `wrangler secret put ADMIN_API_KEY --env production`) || Staging | `wrangler secret` (set via `wrangler secret put ADMIN_API_KEY --env staging`) || Local dev | `worker/.dev.vars` → `ADMIN_API_KEY=...` || Tests | `worker/vitest.config.ts` miniflare bindings |Base URLs: production `https://api.savecraft.gg`, staging `https://staging-api.savecraft.gg`, local `http://localhost:8787`.## Endpoints### Discovery (D1)| Endpoint | Returns ||----------|---------|| `GET /admin/sources` | All sources: `source_uuid`, `user_uuid`, `hostname`, `source_kind`, timestamps. Capped at 200. || `GET /admin/source/:uuid/events?limit=N` | D1 `source_events` rows (default 50, max 500). Includes `event_type`, `event_data`, `created_at`. |### SourceHub Debug (per-source DO)| Endpoint | Returns ||----------|---------|| `GET /admin/source/:uuid/debug/state` | `sourceState` (sources array with online/lastSeen/games), `sourceUuid`, `userUuid`, `sourceMeta`, `alarm` timestamp || `GET /admin/source/:uuid/debug/connections` | `daemonCount`, `connections` array with tags per socket || `GET /admin/source/:uuid/debug/log?level=X&limit=N` | Ring buffer entries (newest first). Levels: `debug`, `info`, `warn`, `error`. Limit capped at 200. || `GET /admin/source/:uuid/debug/storage` | All DO transactional storage key names |### UserHub Debug (per-user DO)| Endpoint | Returns ||----------|---------|| `GET /admin/user/:uuid/debug/state` | `mergedState` (all sources merged), `userUuid` || `GET /admin/user/:uuid/debug/connections` | `uiCount` (active UI WebSocket count) || `GET /admin/user/:uuid/debug/log?level=X&limit=N` | Ring buffer entries. Same filters as SourceHub. || `GET /admin/user/:uuid/debug/storage` | All DO storage key names |## Two Data Sources| Source | What it captures | Lifetime | Use for ||--------|-----------------|----------|---------|| **Ring buffer** (`/debug/log`) | All DO activity: connections, mutations, forwarding, errors | In-memory, 200 entries, lost on DO eviction/deploy | Live debugging, recent activity || **D1 events** (`/events`) | Protocol events + `internalError` catch-block failures | Persistent, last 100 per source | Post-mortem, historical patterns |Ring buffer entries also emit to `console.log` as structured JSON — use `wrangler tail --env production --format json` for real-time streaming.## Debugging Workflows### Source not showing online```1. GET /admin/sources → find source_uuid2. GET /admin/source/:uuid/debug/connections → daemonCount: 0? Daemon WS never connected or was dropped. Check daemon logs for connection errors. → daemonCount: 1+? DO thinks daemon is connected. Continue.3. GET /admin/source/:uuid/debug/state → sources[].online: false? State mutation failed. Check /debug/log?level=error for "state mutation failed". → userUuid: null? SourceHub doesn't know which user owns this source. Source may not be linked. Check D1: source has user_uuid?4. GET /admin/user/:uuid/debug/state → Source missing from mergedState? SourceHub→UserHub forwarding broken. Check SourceHub /debug/log for "forward state to UserHub failed".5. GET /admin/user/:uuid/debug/connections → uiCount: 0? No browser connected. UI issue, not DO issue.```### Events not reaching UI```1. GET /admin/source/:uuid/debug/log → Look for "message received" entries — daemon IS sending events. → Look for "forwarding event to UI" — SourceHub IS forwarding to UserHub.2. GET /admin/source/:uuid/debug/state → check userUuid is set3. GET /admin/user/:uuid/debug/log → Look for "forwarding event to UI" with uiCount > 0. → uiCount: 0 means no UI client to receive.4. GET /admin/source/:uuid/events → Events in D1? Then persistence works but forwarding doesn't. → No events? Check SKIP_PERSIST set — some event types are intentionally skipped.```### Internal errors / silent failures```1. GET /admin/source/:uuid/debug/log?level=error → Shows all catch-block errors with context2. GET /admin/source/:uuid/events?limit=50 → Filter for event_type = "internalError" → Each has: context (which method), error message, stack trace3. Common internal errors: - "forward state to UserHub failed" → UserHub DO unreachable - "event persistence failed" → D1 write error (NOT retried to avoid recursion) - "config push failed" → D1 source_configs query or WS send failed - "state mutation failed" → DO storage read/write error```### Stale source (shows online but daemon is gone)```1. GET /admin/source/:uuid/debug/state → Check sources[].lastSeen — how old? → Check alarm — is the reaper alarm scheduled?2. The alarm fires every 30s (ALARM_INTERVAL_MS). Sources with lastSeen > 90s (STALE_THRESHOLD_MS) get evicted.3. If alarm is null and source shows online → alarm was deleted prematurely. Check /debug/log for "alarm fired" / "alarm rescheduled" entries.```## Security- API key comparison uses `crypto.subtle.timingSafeEqual` (not `===`)- Debug subpaths are allowlisted: only `state`, `connections`, `log`, `storage`- Admin endpoints are read-only — no state mutation possible through them- The admin router is in `worker/src/admin.ts`

joshsymonds/savecraft.gg/tree/main/.claude/skills/debugging-durable-objects commit a7949218a1

Frequently asked questions

npx skillmds@latest add joshsymonds/debugging-durable-objects