Troubleshoot

Troubleshoot Skill

Use this skill to diagnose problems in the Real-Time Voice Agent pipeline. This is a read-only investigative workflow: gather evidence, reason about it, ask the user for the details you don't have, then hand back a clear findings + recommended-fix report.

🧭 Golden rule: Diagnose, don't change. You find and explain the problem. The user decides whether and how to apply the fix.

🚧 Guardrails (read this first)

You MUST NOT (without explicit, per-action user approval):

• ❌ Edit code, YAML, .env*, Terraform, Bicep, or any config file
• ❌ Run state-changing Azure/infra commands: azd up, azd provision, azd deploy, azd down, azd env set, terraform apply/destroy, az ... create/update/delete, az containerapp update, scaling, restarts, key rotation
• ❌ Run git commit, git push, git reset, branch deletes, or bd writes
• ❌ Restart, redeploy, purge, or "nuke and redeploy" anything as a shortcut
• ❌ Invoke any MCP tool with a write side-effect (e.g. anything that creates GitHub issues, updates resources, or scales/restarts) — including a "scan" tool whose options default to writing
• ❌ Print secrets/connection strings/keys in full — mask them (endpoint=...;accesskey=***)

You MUST:

• ✅ Prefer read-only commands (show, list, get, logs, query, curl, ping)
• ✅ Ask before assuming — see Step 0. Probe for the missing detail instead of guessing
• ✅ Present a fix as a recommendation the user applies, or ask for approval before running it
• ✅ State your confidence and what evidence is still missing

If a fix genuinely requires a write action, stop and propose it — show the exact command, explain the blast radius, and let the user run it or approve it.

Step 0 — Ask before you assume

Never start guessing. First pin down the situation with a few targeted questions. Ask only what you can't already infer from the repo or the user's message:

1. Where is it failing? Local dev (make start_backend) or deployed (Container Apps via azd)?
2. Which environment? (azd env list shows them — confirm which one.)
3. What's the symptom, concretely? Error text, no audio, dropped call, high latency, wrong agent, silence, garbled speech, 4xx/5xx, timeout?
4. What changed recently? New deploy, config edit, region, model, phone number, dependency bump?
5. Reproducibility? Every call or intermittent? A specific agent/scenario/number?
6. What do you already have? Logs, a call_id/correlation id, screenshots, the azd env values?

If the user gives a vague report ("calls don't work"), ask for one concrete failing example (a call_id, a timestamp, the exact error) before diving in. One good example beats ten guesses.

Triage map — symptom → likely layer

Read-only diagnostic playbook

Azure MCP (preferred fast path for deployed envs)

If an Azure SRE-agent MCP server is wired into this workspace, prefer its purpose-built tools over hand-rolled CLI — they already know the ARTagent topology. Confirm they're available first (don't assume; if no Azure MCP is connected, fall back to the az/curl commands below):

⚠️ analyze_voice_channel_security defaults auto_create_issues: true — that creates GitHub issues (a write action). Under these guardrails you must call it with auto_create_issues: false, or get explicit user approval before letting it file issues.

Generic Azure MCP servers (resource lookup, az-equivalent, App Insights KQL) are fine too — use only their read operations (get/list/show/query). Never invoke create/update/delete MCP operations as part of diagnosis. If you need historical KQL/latency analysis, hand off to the observability-insights skill.

A. Local dev

# Is the backend up and what does it think is healthy?
curl -s http://localhost:8010/api/v1/health        | jq .
curl -s http://localhost:8010/api/v1/readiness     | jq '{status, checks}'
curl -s http://localhost:8010/api/v1/pools         | jq .
curl -s http://localhost:8010/api/v1/metrics/summary | jq .

# Port already taken? (inspect only — do NOT kill without asking)
lsof -iTCP:8010 -sTCP:LISTEN -n -P

# What config does the app actually see? (mask before sharing)
azd env get-values 2>/dev/null | sed -E 's/(accesskey|key|password|secret)=[^;]*/\1=***/gi'

B. Deployed (Container Apps)

# Resolve the deployed backend from azd artifacts
azd env select <env>
BACKEND="https://$(azd env get-value BACKEND_CONTAINER_APP_FQDN)"

# Health across all dependencies (read-only)
curl -s --max-time 10 "$BACKEND/api/v1/readiness" | jq '{status, checks}'

# Or run the bundled script
./devops/scripts/quick_health_check.sh <env>

# Live + recent logs (read-only)
az containerapp logs show --name <app> --resource-group <rg> --follow      # tail
az containerapp logs show --name <app> --resource-group <rg> --tail 200    # snapshot
az containerapp revision list --name <app> --resource-group <rg> -o table  # which revision is live

C. Per-layer probes

# Redis / state
make test_redis_connection          # connectivity only
make connect_redis                  # interactive inspect (read keys; don't FLUSH)

# App Config (what runtime values resolve to)
make show_appconfig
make show_appconfig_acs

# Azure OpenAI quota / 429 cause (read-only)
az cognitiveservices account deployment list -g <rg> -n <openai-account> -o table
az cognitiveservices usage list -l <region> -o json \
  | jq -r '.[] | select(.name.value | startswith("OpenAI.")) | "\(.name.value)\t\(.currentValue)/\(.limit)"'

When the problem needs historical traces, latency percentiles, or a call timeline across the deployed stack, switch to the observability-insights skill — it builds the wider picture from Azure Monitor / Log Analytics and renders it as KQL + diagrams.

How to reason

1. Confirm the layer using the triage map and the health/readiness output (don't assume).
2. Pull evidence with read-only commands for that layer only — avoid shotgun-running everything.
3. Correlate by call.connection.id (ACS), session.id (browser), or operation_Id across logs, traces, and the user's report — App Insights has no call_id field.
4. Form one hypothesis at a time, name the evidence for and against it, and the gap.
5. If evidence is missing, ask the user for the specific artifact rather than assuming.

Output format

End every investigation with this structure:

### Diagnosis
- **Symptom:** <what the user observed>
- **Most likely cause:** <layer + root cause> (confidence: high/medium/low)
- **Evidence:** <commands run + key lines, secrets masked>
- **Still unknown:** <what you'd need to confirm>

### Recommended fix (you apply this)
1. <exact command or file edit, shown — not executed without approval>
2. <verification step to confirm it worked>

### If that doesn't resolve it
- <next hypothesis to test>

Reference

• Quick fixes by error: TROUBLESHOOTING.md
• Deploy flow & hooks: deployment-guide skill
• Wider context + visuals: observability-insights skill
• Azure MCP fast path (when connected): SRE-agent check_deployment_health, analyze_deployment_logs, analyze_pool_metrics, analyze_voice_channel_security (set auto_create_issues: false)
• Health endpoints: apps/artagent/backend/api/v1/endpoints/health.py
• Health script: devops/scripts/quick_health_check.sh