Tt Studio Debug Bundle

TT-Studio Debug-Bundle Triage

When this fires

User shares any of:

• A path like /.../tt-studio-logs-ttbr-<hex>/ or a ZIP with the same name
• A ttbr-<hex> reference (the diagnostic ID auto-generated by the TT-Studio Bug Reporter)
• A screenshot of GitHub issue from tenstorrent/tt-studio whose title contains "TT-Studio bug report"
• Log lines like Allocation failed for ...: Invalid device_id, ChipSlotAllocator initialized: board=, Raw board_type:

Bundle layout (assume this shape)

tt-studio-logs-ttbr-<hex>/
├── agent.log                       # AI-agent service (LangGraph, voice, etc.)
├── backend.log                     # Django backend — the main signal
├── current_models.json             # Live docker containers + deploy_cache snapshot
├── deployments.json                # Persisted deploy records (id, device, device_ids, status)
├── docker-control-service.log      # docker daemon control
├── model_run.log                   # FastAPI inference-control server
├── model_run_logs/                 # Per-deploy model run logs:
│   └── model_run_<ts>_<model>_<device>_server.log
├── inference_artifacts/
│   ├── docker_server/              # Container stdout per deploy. Two flavors:
│   │   ├── vllm_<ts>_<model>_<device>_server.log    # LLMs (Llama, Qwen, …)
│   │   └── media_<ts>_<model>_<device>_server.log   # TTS / ASR / Whisper
│   │                               #   (speecht5_tts, distil-large-v3, …)
│   └── run_logs/                   # tt-inference-server run.py logs (one per deploy)
├── startup.log
└── tt_smi.json                     # Raw hardware enumeration from tt-smi

Workflow

Copy this checklist and tick as you go:

Triage Progress:
- [ ] 1. Inventory the bundle
- [ ] 2. Read the three metadata JSONs
- [ ] 3. Pull the smoking-gun lines from backend.log
- [ ] 4. Read the failing deploy's fastapi + inference logs
- [ ] 5. Fan out parallel Explore subagents on source
- [ ] 6. Synthesize the chain (hardware → detection → allocation → request → user-visible error)
- [ ] 7. Report root cause + minimal fix shape

1. Inventory

ls -la <bundle>/ <bundle>/model_run_logs <bundle>/inference_artifacts/docker_server <bundle>/inference_artifacts/run_logs

Note: which models have logs, which deploys are most recent, whether current_models.json shows live containers or an empty deploy_cache_entries.

2. Metadata JSONs — what to extract

Critical cross-check: same board_id across multiple device_info entries = one physical card, multiple ASICs. Different board_ids = multiple cards. The num_devices count and board_type together determine which DeviceConfigurations enum the backend picks.

3. backend.log keyword sweep

grep -nE 'Raw board_type|Detected.* board type|ChipSlotAllocator initialized|Allocation failed|Manual allocation|Invalid device_id|is_compatible|compatible=False, boards' <bundle>/backend.log | head -80

Read those lines. They reveal:

• What board_type the backend cached (and the slot count it chose)
• Which models were marked incompatible and why (the boards list)
• The exact device_id rejections with the valid range (Must be 0-N)
• Which model the user actually got to deploy

4. Failing-deploy logs

For each FAILED deploy in deployments.json, read the matching model_run_logs/model_run_<ts>_<model>_<device>_server.log to capture the exact run.py command line (model, workflow, device, device-id flags). This is the request the backend produced.

If a container log exists under inference_artifacts/docker_server/, grep it for Traceback, Error, OOM, Failed, tt:: and read context around the first hit. Pick the right file by model_type from current_models.json:deploy_cache_entries[].model_impl.model_type:

• vllm_<ts>_<model>_<device>_server.log — LLMs served by vLLM (model_type empty/llm/generation). Traceback patterns: EngineCore failed to start, _set_hf_params, AutoConfig.from_pretrained.
• media_<ts>_<model>_<device>_server.log — media-inference-server for TTS (speecht5_tts), ASR/STT (distil-large-v3, whisper variants). model_type is tts or speech_recognition. Traceback patterns: container-side tt:: device errors, model-spec load failures, audio pipeline errors. The Voice Agent pipeline depends on both an LLM (vllm log) AND two media services (two media logs) — if the user reports a Voice Agent failure, check all three.

Run logs (inference_artifacts/run_logs/run_<ts>_id_<impl>_<model>_<device>_server_<hash>.log) wrap both flavors and contain the docker run invocation (mounts, device passthrough, volume name) regardless of whether it's vLLM or media-server.

Triage gate before fan-out: if the container log (vllm_*** or media_***) shows Fabric initialized on N devices / device init succeeded, and the crash happens after that (inside model loading — AutoConfig.from_pretrained, _set_hf_params, missing tokenizer files, missing audio model files, bad weight shapes), the bug is not in the backend allocator / detection / compat code. It's a model-asset problem inside the persistent docker volume (volume_id_<impl>-<model>). Skip step 5 entirely and jump to the remediation playbook — the chain is hardware ✅ → allocation ✅ → container launch ✅ → model files ❌. Applies to both vLLM and media-server containers.

5. Fan out — parallel Explore subagents

Do this in a single message with 3–4 parallel Agent calls using subagent_type: Explore. Brief each one with:

• The specific log evidence (paste the exact lines)
• The specific source files they should read
• A demand for file:line citations and quoted code

Default fan-out targets (adjust per symptom):

6. Synthesize the chain

Always express the result as a chain, with each link cited:

1. tt-smi reports: <facts from tt_smi.json>
2. Backend normalizes to: <enum> (services.py:<line>)
3. Slot count chosen: <N> (chip_allocator.py:<line>)
4. Frontend requested: device_ids=<list> (<component>.tsx:<line>)
5. Backend rejected: <error> (chip_allocator.py:<line>)
6. User saw: <UI symptom from screenshot / bug report>

This makes the broken link obvious.

7. Report

End with:

• Root cause in one sentence
• Smallest fix — the single config/dict change or branch that would unblock, with file:line
• What it doesn't fix — call out unresolved UX gaps (e.g. "Voice Agent still needs 3 chips; a 2-ASIC P300 can't run the full pipeline even with the slot fix")

Don't propose patches unless the user asks. Stop after the report.

Common failure signatures

Remediation playbook — corrupt model volume

When the failure is a model-asset problem (signatures above), the fix is not a code change — delete the volume and redeploy. Tell the user:

1. Verify the volume is the problem (cheap, read-only):

   docker run --rm -v volume_id_<impl>-<model>:/cache alpine \
     sh -c 'ls -la /cache/model_file_symlinks_map/<model>/ && cat /cache/model_file_symlinks_map/<model>/config.json 2>&1 | head -20'
   

   Expect either No such file or directory for config.json, or a file missing the "model_type" key. The volume name format is volume_id_<impl-with-underscores>-<model-name> — find it in the --volume … line of the failing deploy's run.py log under inference_artifacts/run_logs/.

2. Stop the dead container (if still present in current_models.json / docker ps -a):

   docker rm -f <container_name>
   

3. Delete the volume so run.py will re-fetch and re-symlink the model on next deploy:

   docker volume rm volume_id_<impl>-<model>
   

4. Redeploy from the TT-Studio UI. Confirm HF_TOKEN is set in .env (the run.py log will say HF_TOKEN is set); if not, gated weights won't download.

Call out as an unresolved UX gap in the report: the backend currently polls a dead container for hours emitting Health check failed: 405: Model is not ready (/backend/model_control/model_utils.py:176) without surfacing the vLLM Engine core initialization failed traceback to the user — so this failure looks like an endless spinner. Worth flagging for a separate issue.

Anti-patterns

• Don't start reading source code before you've read the JSON snapshots and grep'd backend.log — the logs tell you which code path to read.
• Don't run the subagents sequentially. They're independent — single message, multiple Agent blocks.
• Don't speculate about hardware state from screenshots alone; always ground claims in tt_smi.json.
• Don't write a fix patch unless asked. The user usually wants the diagnosis first.
• Don't assume every failure is a backend/allocator bug. If Fabric initialized succeeded and the crash is inside HF model loading, it's a volume/asset problem — recommend docker volume rm + redeploy, not source code reading.