# Incident Investigation

> Use this skill to diagnose and resolve production incidents in VoxBento.

- Skill: `fossasia/incident-investigation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add fossasia/incident-investigation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/fossasia/incident-investigation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: fossasia (https://skillmd.com/u/fossasia)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/fossasia/incident-investigation

---


# Skill: Incident Investigation

> Use this skill to diagnose and resolve production incidents in VoxBento.

---

## Diagnostic Entry Points

### 1. Is the portal up?
```bash
curl http://localhost:8000/healthz
# Expected: {"ok": true, "server": "fastapi", "mediamtx_ok": true}
```

### 2. Is MediaMTX up?
```bash
curl http://localhost:9997/v3/paths/list
# Expected: 200 with paths array
docker-compose ps mediamtx
docker-compose logs mediamtx --tail=50
```

### 3. Portal logs
```bash
docker-compose logs portal --tail=100
# Look for: ERROR, WebSocketDisconnect, DB errors, transcription errors
```

### 4. Active booth state (no API for this — check portal logs or DB)
```bash
# Check live booth count from admin panel: /admin/
# Or query DB directly:
docker-compose exec portal uv run python -c "
import asyncio
from portal.database import get_session, list_events
async def main():
    async with get_session() as s:
        events = await list_events(s)
        for e in events:
            print(e.slug, e.display_name)
asyncio.run(main())
"
```

---

## Common Incident Types

### IC-01: Interpreter cannot "Go Live"

**Symptoms:** "Go Live" button clicked but WHIP fails or spinner never resolves.

**Diagnosis:**
1. Check browser DevTools Network tab for `GET /api/events/{slug}/booths/{lang}/whip-url`.
   - 403? → Interpreter is not the active interpreter. Check `booth:state` WS message.
   - 404? → Booth not in memory. Was the WS `booth:join` message sent successfully?
2. Check MediaMTX reachability: `/healthz` → `mediamtx_ok`.
3. Check `MEDIAMTX_WHIP_BASE` is browser-reachable (not Docker-internal URL).
4. Check WebSocket connection in DevTools → WS tab.

**Fix:** If not active interpreter: coordinator must reassign via `booth:set-active`. If MediaMTX down: `docker-compose restart mediamtx`.

---

### IC-02: Listener hears nothing

**Symptoms:** Listener page loads but WHEP connection shows "disconnected" or audio is silent.

**Diagnosis:**
1. Check interpreter is "Go Live" — `ingest_status == 'connected'` in booth state.
2. Check `MEDIAMTX_WHIP_BASE` in listener page source — must be browser-reachable HTTPS.
3. Browser DevTools → check RTCPeerConnection state in `whep-listener.js`.
4. Check MediaMTX path: `GET http://localhost:9997/v3/paths/get/{event_slug}/{language_code}`.
5. Check `alwaysAvailable: true` on the path — if not set, WHEP fails when no publisher.

**Fix:** Ensure `_ensure_mediamtx_path` was called. Manually: `PATCH http://mediamtx:9997/v3/config/paths/patch/{path}` with `{"alwaysAvailable": true}`.

---

### IC-03: WebSocket disconnects repeatedly

**Symptoms:** Interpreters see connection status flickering; participants disappear and reappear.

**Diagnosis:**
1. Check portal logs for `WebSocketDisconnect`.
2. Check browser DevTools WS tab for close code:
   - 4001: Missing/invalid token.
   - 4003: Token scope mismatch.
   - 1006: Abnormal closure (network issue or portal crash).
3. Check for `asyncio.Lock` deadlock in `portal/booth_state.py` — portal becomes unresponsive.
4. Check for uncaught exceptions in `_handle_*` functions (portal logs).

**Fix:** Restart portal if deadlocked. Fix token scope if 4003 (token was generated for different booth).

---

### IC-04: Transcription not appearing

**Symptoms:** Booth is live, transcription enabled, but no captions appear.

**Diagnosis:**
1. Check `active_workers` state — portal logs should show worker start: `Starting {provider} transcription worker for booth {booth_id}`.
2. Check ffmpeg can reach MediaMTX RTSP: `docker-compose exec portal ffmpeg -rtsp_transport tcp -i rtsp://mediamtx:8554/{event_slug}/{language_code} -f null - -t 5`.
3. Check API key exists and is valid: portal logs for `API key missing` or `Failed to decrypt`.
4. Check `event.transcription_api_enabled` is `True` for external providers.
5. Check `MAX_TOTAL_WORKERS` (10) not exceeded: count workers in portal logs.
6. Check booth DB config: `db_booth.transcription_enabled`, `db_booth.transcription_provider`, `db_booth.transcription_model`.

**Fix:** Restart transcription worker via admin panel → booth detail → transcription settings (re-save). Or call `stop_transcription_worker(booth_id)` + `start_transcription_worker(...)` via debug endpoint if available.

---

### IC-05: Admin login fails

**Symptoms:** `/admin/login` with correct password → still shows error.

**Diagnosis:**
1. Check `ADMIN_PASSWORD` env var is set and not empty.
2. Check `admin_token` cookie is being set (DevTools → Application → Cookies).
3. Check JWT secret is consistent (`settings.effective_jwt_secret`).
4. Alternative: log in as user with `is_admin=True` — uses `/login` + `user_token` cookie.

---

### IC-06: Database errors

**Symptoms:** 500 errors on any page involving DB; portal logs show SQLAlchemy errors.

**Diagnosis:**
1. Check DB URL: `DATABASE_URL` env var.
2. Check migrations are current: `uv run alembic current` should show `head`.
3. If SQLite: check `portal-data` volume is mounted and has write permission.
4. If PostgreSQL: check connection string and DB server availability.

**Fix:** Run `uv run alembic upgrade head`. If corrupt SQLite: restore from backup volume.

---

## Emergency Commands

```bash
# Restart portal only
docker-compose restart portal

# View live logs
docker-compose logs portal -f

# Force-stop all services
docker-compose down

# Full restart
docker-compose up -d

# Apply pending migrations manually
docker-compose exec portal uv run alembic upgrade head

# Check DB migration state
docker-compose exec portal uv run alembic current
```