Debug Companions Prod

Debug Companions production

Operator-machine skill for diagnosing the production Companions Runtime v2 stack: four Railway services (web, api, worker, runtime), PostgreSQL, and box.ascii.dev Boxes running Pi. Everything here is read-only except railway_restart.py, which is double-gated and never touches release.

Authoritative references: docs/runbooks/companions-runtime.md (operations), docs/companions-runtime.md (state machine), deploy/railway/README.md (credential boundary).

Hard safety rules

From the runbook — these are not negotiable:

1. Never paste tokens, signed URLs, provider payloads, raw Pi lines, auth files, or decrypted material into a transcript, ticket, or log search. Every script pipes output through redact(); do not bypass the scripts to run raw curl/psql with credentials in argv.
2. Never delete or archive a Box. Full Box restart is an explicit user action only; automatic repair may recycle Pi but never a healthy Box. This skill ships no Box mutation code at all.
3. Never manually mark an ambiguous or interrupted attempt queued, clear lease rows, or edit epochs. An ambiguous dispatch is resolved only by an Owner/Editor Retry (new retry_id) or Cancel in the product.
4. The kill switch (companion_runtime_disable) is a human, migration-owner action. This skill only reads gate status via db_query.py gate; it must never call enable/disable.
5. Queued durable work is not garbage. Do not delete rows to make a dashboard look clear.
6. PROD_DATABASE_READ_URL must reference a read-only login. The scripts still wrap every query in BEGIN TRANSACTION READ ONLY; ... ROLLBACK; as defense in depth, and have no free-SQL mode.

Prerequisites

• psql on PATH (PostgreSQL 17 client).

• ~/.companion-prod.env, mode exactly 0600 (scripts refuse otherwise):

  RAILWAY_API_TOKEN=...            # Railway token (bearer or project token)
  RAILWAY_PROJECT_ID=...
  RAILWAY_ENVIRONMENT_ID=...
  COMPANION_BOX_API_KEY=...        # box.ascii.dev key — runtime-only elsewhere
  COMPANION_BOX_API_BASE=https://ascii.dev/api/box/v1   # optional, this default
  PROD_DATABASE_READ_URL=postgres://...                 # READ-ONLY role
  DEBUG_PROD_ALLOW_RESTART=0       # set to 1 only while deliberately restarting
  

  chmod 600 ~/.companion-prod.env
  

Values are read only from this file, passed to subprocesses via the environment, and never printed or placed in argv.

First five minutes

Run from the repository root:

python3 .claude/skills/debug-companions-prod/scripts/railway_status.py
python3 .claude/skills/debug-companions-prod/scripts/db_query.py gate
python3 .claude/skills/debug-companions-prod/scripts/db_query.py health
python3 .claude/skills/debug-companions-prod/scripts/railway_logs.py \
  --service runtime --since 30m --grep '"level":"error"|persisted|denial' --raw

Interpretation order:

1. railway_status — are all four services on the same commit and SUCCESS? A commit mismatch during an incident usually means a half-finished deploy.
2. gate — enabled=false means claims are fenced (kill switch or cutover in progress); nothing will run until a human re-enables with the observed epoch. Do not "fix" this from the skill.
3. health — queued backlog per companion, active attempts, stale instance heartbeats, expired-but-claimed leases.
4. Runtime error logs — collect stable code values, then follow the symptom playbooks below and references/triage-playbook.md.

Symptom playbooks

Box launches are failing

python3 scripts/db_query.py interrupted --since 6h
python3 scripts/db_query.py ops --companion <uuid>
python3 scripts/db_query.py instance --companion <uuid>
python3 scripts/box_list.py --companion <uuid>
python3 scripts/railway_logs.py --service runtime --companion <uuid> --since 6h

• cold_start_deadline_exceeded on turns + operations stuck around creating_box/waiting_ready/installing_layout: the cold path (Box ready
  • Pi install) exceeded the 3-minute SQL deadline. Check ops checkpoints and attempt_count to see where time went.
• A cold_start_deadline_exceeded operation whose started_at is already later than the turn deadline, with no attempt and an existing warm idle Box, is the pre-0129 queued-follow-up bug: the send was misclassified while Pi was busy. It is not evidence of a Box cold start.
• box_create_ambiguous: create may have committed provider-side. Run box_list.py --companion — two Boxes with the same generation is the evidence. Do NOT delete either; the runtime discovers the generation-qualified name and selects one canonical Box (runbook: Box lifecycle/provider outage).
• box_rate_limited (429) or box_provider_unavailable/box_network_error: provider incident; count occurrences over the window before escalating to ascii.dev.

Chat dies or stalls while waiting for input

Three distinct signatures can produce "my chat died while I was away". Identify which one you have before touching anything:

python3 scripts/db_query.py turn --turn <uuid>
python3 scripts/db_query.py decisions --companion <uuid> --since 24h
python3 scripts/railway_logs.py --service runtime --turn <uuid> --since 24h

1. Decision expiry (ask_user timeout, 10 minutes). decisions shows a question/confirmation row with decision_status=expired and an expires_at roughly ten minutes after creation; cancelled before that can mean a newer member message returned control to Pi. Neither state grants approval. On releases before migration 0129, a decision can instead be followed by turn_stalled after ten minutes because the inactivity clock was not actually paused.
2. pi_event_stream_interrupted. The attempt's error triplet names this code: the broker's event stream from Pi broke mid-turn. Look at the attempt's unknown_event_count/malformed_event_count and runtime logs around last_activity_at. This is transport loss, not member behavior; frequent occurrences are the provider-polling failure mode the direct transport work targets.
3. turn_stalled (10-minute inactivity). Terminal status interrupted with code turn_stalled: Pi acknowledged the attempt but produced no correlated activity for ten minutes. Distinguish from case 1 by the absence of a pending/expired decision row; distinguish from case 2 by the error code. Check whether Pi is wedged (instance shows pi_state) — Retry recycles Pi.

The decision expiry and the ten-minute running stall are different clocks. After migration 0129, needs_input pauses inactivity; before it, trust the row timestamps and error code over the expected state-machine semantics.

Turn interrupted or Pi silent

python3 scripts/db_query.py stuck
python3 scripts/db_query.py turn --turn <uuid>

• dispatch_state=ambiguous / prompt_dispatch_ambiguous: the prompt may have reached Pi; it is deliberately never replayed. The queue is blocked until an Owner/Editor retries or cancels — that is the design, not a bug. Warn that earlier external effects may have succeeded.
• An interrupted queue head with queued turns behind it (stuck) needs an Owner/Editor decision in the product, not a database edit.
• turn_deadline_exceeded: two-hour absolute deadline; look at attempt history for what consumed it.
• Retry creates a new attempt and recycles Pi; it never restarts the Box.

/healthz unhealthy (503)

Runtime healthz is private; diagnose via logs and the database, not curl:

python3 scripts/railway_logs.py --service runtime --since 15m --raw
python3 scripts/db_query.py health

• database=false: private database path or restricted runtime login broken. Never substitute the API or owner URL.
• claim_loop=false: preserve the first stable error code from logs, then roll one replica (railway_restart.py, double-gated). If another replica cannot take over within 45 seconds, the kill switch is a human decision.
• sweep_fresh=false: event-loop starvation or stuck sweep. A process still accepting TCP is not healthy; roll it.

Error-code map

Stable codes from packages/companion-runtime/src/errors.ts and the runtime adapters (persisted triplet: code, expurgated ≤500-char message, action):

outbox_harvest_failed is a process-log event, not a persisted attempt error: the turn succeeded and only reply images were partially recovered — search logs, never reclassify the turn.

Escalation

• Provider-side (create failures, 429 storms, Boxes stuck provisioning, resume instability): collect Box ids, timestamps, and counts (redacted output only) and escalate to ascii.dev.
• Kill-switch-worthy (unsafe duplicate execution, credential exposure, broken fencing, corrupt projection): stop; page the on-call owner. The fence is companion_runtime_disable(<observed_epoch>, 'incident-<id>') run by the migration owner — not by this skill.
• Suspected secret exposure: fence first, then rotate per runbook (Box key on runtime only, desktop HMAC on api+runtime together). Search logs only for stable identifiers and codes.
• Record environment, release commit, and operator/change id for every production change. Never record secret values.

references/triage-playbook.md has the full symptom → evidence → cause → runbook-section map; references/railway-api.md documents the (UNVERIFIED) GraphQL surface; references/redaction.md is the redaction contract.