Debug Voice

Debug Voice

Make a voice session readable after the fact, then fix from evidence. The agent cannot hear the app; the log is its ears, the user is its judge. No audio, no tokens, never in prod.

Works for any stack. The pipeline is a small contract (below); implement it in whatever the app already uses.

Workflow

1. Map the app (read only).
2. Plan: write the change list, show it, stop. Nothing is edited until the user aligns.
3. Install the agreed pieces in the app's language, framework, and conventions.
4. Verify the sink, hand the app to the user.
5. Fix loop from the log.

1. Map

Find, and note the paths:

• Voice client: where realtime events are received and sent, mic capture, audio playback, token fetch.
• Server: framework, how routes are declared, where shared server code lives, how env is read, what "production" means here.
• Conventions: language(s), module system, formatter, where scripts or tasks live (package.json, Makefile, pyproject, justfile), .gitignore.
• Client kind: browser, mobile, desktop, CLI. A non-browser client still POSTs the same JSON; a single-process app can skip HTTP and append to the file directly.
• Where audio deltas are handled. They must be counted, never logged.

2. Plan, then stop

Fill this in with real paths and the app's language, post it, and wait for a yes or a trimmed list. Do not edit files before that.

## Debug voice: plan

Add
- <path>: client logger (batch, redact, flush) in <language>
- <path>: dev-only sink `POST /api/voice/log` → `.voice-logs/<sessionId>.ndjson`
- <path> (optional): summary command `<cmd>`; otherwise read the NDJSON with jq

Modify
- <voice client file>: hook points start, token, mic, env, ws.*, client/server events,
  audio.in (2 s windows), audio.out.first, audio.out, play.stop, stop
- <token route>: append `server.token { ok, status, ms, upstream }` (never the token)
- <UI file>: session id in the voice status line and in voice error messages
- .gitignore: `/.voice-logs`
- <scripts file>: a `voice:logs` task (only if the summary command is wanted)

Logged: event names and non-audio fields, timings, byte counts, mic RMS.
Never: tokens, API keys, raw audio, strings over 400 chars.
Off in production unless `VOICE_LOG=1`.

Reply "go", or strike lines you do not want.

3. Install: the contract

Match the app. Same language as the surrounding code, same route style, same formatter. Write the pieces from the contract below; do not introduce a second language or toolchain for logging.

Session id: 8 lowercase hex chars from a UUID. The sink accepts ^[a-z0-9]{4,64}$; it becomes a file name.

Entry (one JSON object per line):

Redaction, applied client side before buffering: on audio event types (response.output_audio.delta, response.audio.delta, input_audio_buffer.append) replace delta / audio with bytes = decoded base64 length; strings over 400 chars cut to 400 + …[N chars]; objects deeper than 4 → "[depth]"; arrays over 50 items truncated.

Client logger: buffer entries; flush as POST <sink> {"sessionId","entries":[…]} every 1 s or at 200 entries; on stop flush with keepalive (or the platform's "survive navigation" equivalent); swallow every transport error, logging must never throw into the voice path. Also mirror entries to the console in dev.

Sink: POST /api/voice/log, JSON body. 404 unless dev or VOICE_LOG=1. 400 if sessionId fails the regex or entries is not an array. Append at most 500 entries per request, drop any line over 16,000 chars, to .voice-logs/<sessionId>.ndjson, creating the directory. Reply 204.

Pseudocode for any server:

handle POST /api/voice/log:
  if production and VOICE_LOG != "1": return 404
  body = parse json or return 400
  if not regex(body.sessionId) or not list(body.entries): return 400
  mkdir .voice-logs; append join(json(e) for e in body.entries[:500] if len < 16000) to .voice-logs/{sessionId}.ndjson
  return 204

Pseudocode for the client logger:

logger(sessionId, sink):
  buffer = []; started = now()
  log(kind, data): buffer.push({ ...redact(data), t: now() - started, ts: epoch_ms(), kind }); schedule flush (1 s timer, or immediately at 200 entries)
  server(event, extra): log("server", { ...redact_event(event), ...extra })   # never per audio delta
  client(event):        log("client", redact_event(event))                    # never per audio chunk
  error(where, err, extra): log("error", { where, name, message, ...extra })
  flush(final=false): POST sink {"sessionId","entries": buffer}; buffer = []; ignore all errors; keepalive when final
  close(): flush(final=true)

Hook points

kind and fields; the shape is the same in every language.

In the message handler: if the event is an audio delta, count bytes and gaps and play it; otherwise log it as server with the current phase, then run the existing handling. Keep speechStoppedT from input_audio_buffer.speech_stopped and createdT from response.created; report both distances on the first delta. In the player, count an underrun when the next scheduled time is already in the past mid-response, track the largest drain, reset on response.created.

UI: show the id in the voice status line (Listening · session ab12cd34) and append (voice session <id>) to voice errors, so the user can name the run.

4. Verify

curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:<port>/api/voice/log \
  -H 'Content-Type: application/json' \
  -d '{"sessionId":"smoke001","entries":[{"t":0,"ts":0,"kind":"start"}]}'   # 204
curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:<port>/api/voice/log \
  -H 'Content-Type: application/json' -d '{"sessionId":"../x","entries":[]}' # 400
cat .voice-logs/smoke001.ndjson && rm .voice-logs/smoke001.ndjson

Then hand off: the user tests in the real app and reports what they said, what they heard, when it went wrong, and the session id.

Reading a log

Any of these; none needs the app's toolchain:

f=.voice-logs/<id>.ndjson
jq -r 'select(.kind|IN("start","token.ok","mic.ok","env","ws.open","ws.close","server.token","stop","error")) | "\(.t // .ts)ms \(.kind) \(.type // "") \(.message // "")"' $f   # milestones and errors
jq -r 'select(.kind=="server") | .type' $f | sort | uniq -c | sort -rn                      # server event counts
jq -c 'select(.kind|IN("audio.out.first","audio.out"))' $f                                  # per-turn latency, gaps, underruns
jq -c 'select(.kind=="audio.in")' $f                                                        # mic windows, rms

Or read the file; it is one object per line in time order. Server entries have only ts; align them to the client clock with the start entry's ts. If the user wants a summary command, write one in the app's language that prints: milestones, server event counts, one line per audio.out turn with its audio.out.first latencies, mic window totals, errors and closes, and the last 25 entries.

5. Fix loop

1. Instrument (steps 1–4) if the app has no .voice-logs pipeline yet.
2. User tests in the real app and describes the run.
3. Read the log around the failing t.
4. Match symptom → signature → fix (table below). No matching signature: add logging first, re-test, then fix.
5. Fix one thing, re-test, confirm the signature is gone in a fresh log.
6. Write it down: append a confirmed row under "Confirmed from sessions". If the fix changes how voice should be built, update /add-voice too.

Symptom → log signature → fix (starter rows)

Confirmed from sessions

Append after a fix is verified in a fresh log. Format: YYYY-MM-DD · symptom · signature · fix · file(s).

Rules

• Plan first; no edits before the user aligns on the change list.
• Never tokens, keys, or raw audio. Audio becomes byte counts.
• Dev only. The sink returns 404 in production unless VOICE_LOG=1. .voice-logs/ is gitignored.
• Logging never throws into the voice path.
• Aggregate audio; never log per chunk.
• One change per re-test so the log tells you which fix worked.
• Do not invent xAI event names; confirm in https://docs.x.ai/developers/model-capabilities/audio/speech-to-speech before adding a signature.