# Tt Studio Debug Bundle

> Triage a TT-Studio bug-report ZIP / log bundle (named `tt-studio-logs-ttbr-*` or referenced as `ttbr-*`) to root-cause why a model deploy, board allocation, or Voice Agent pipeline failed. Use when the user shares a path to a TT-Studio log bundle, mentions a bug report ID like `ttbr-<hex>`, or attaches a screenshot of the TT-Studio Bug Report issue template. Fans out parallel Explore subagents across `app/backend/board_control/`, `app/backend/docker_control/`, `app/backend/shared_config/`, and `app/frontend/src/components/` to map log evidence to source code.

- Skill: `tenstorrent/tt-studio-debug-bundle` (Agent Skill)
- Install (CLI): `npx skillmds@latest add tenstorrent/tt-studio-debug-bundle`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tenstorrent/tt-studio-debug-bundle/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: tenstorrent (https://skillmd.com/u/tenstorrent)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tenstorrent/tt-studio-debug-bundle

---


# TT-Studio Debug-Bundle Triage

## When this fires

User shares any of:
- A path like `/.../tt-studio-logs-ttbr-<hex>/` or a ZIP with the same name
- A `ttbr-<hex>` reference (the diagnostic ID auto-generated by the TT-Studio Bug Reporter)
- A screenshot of GitHub issue from `tenstorrent/tt-studio` whose title contains "TT-Studio bug report"
- Log lines like `Allocation failed for ...: Invalid device_id`, `ChipSlotAllocator initialized: board=`, `Raw board_type:`

## Bundle layout (assume this shape)

```
tt-studio-logs-ttbr-<hex>/
├── agent.log                       # AI-agent service (LangGraph, voice, etc.)
├── backend.log                     # Django backend — the main signal
├── current_models.json             # Live docker containers + deploy_cache snapshot
├── deployments.json                # Persisted deploy records (id, device, device_ids, status)
├── docker-control-service.log      # docker daemon control
├── model_run.log                   # FastAPI inference-control server
├── model_run_logs/                 # Per-deploy model run logs:
│   └── model_run_<ts>_<model>_<device>_server.log
├── inference_artifacts/
│   ├── docker_server/              # Container stdout per deploy. Two flavors:
│   │   ├── vllm_<ts>_<model>_<device>_server.log    # LLMs (Llama, Qwen, …)
│   │   └── media_<ts>_<model>_<device>_server.log   # TTS / ASR / Whisper
│   │                               #   (speecht5_tts, distil-large-v3, …)
│   └── run_logs/                   # tt-inference-server run.py logs (one per deploy)
├── startup.log
└── tt_smi.json                     # Raw hardware enumeration from tt-smi
```

## Workflow

Copy this checklist and tick as you go:

```
Triage Progress:
- [ ] 1. Inventory the bundle
- [ ] 2. Read the three metadata JSONs
- [ ] 3. Pull the smoking-gun lines from backend.log
- [ ] 4. Read the failing deploy's fastapi + inference logs
- [ ] 5. Fan out parallel Explore subagents on source
- [ ] 6. Synthesize the chain (hardware → detection → allocation → request → user-visible error)
- [ ] 7. Report root cause + minimal fix shape
```

### 1. Inventory

```bash
ls -la <bundle>/ <bundle>/model_run_logs <bundle>/inference_artifacts/docker_server <bundle>/inference_artifacts/run_logs
```

Note: which models have logs, which deploys are most recent, whether `current_models.json` shows live containers or an empty `deploy_cache_entries`.

### 2. Metadata JSONs — what to extract

| File | Pull these fields |
|---|---|
| `tt_smi.json` | `device_info[].board_info.board_type`, `bus_id`, `board_id`, `ASIC_LOCATION`; **count** of `device_info` entries |
| `deployments.json` | `records[].{model_name, device, device_id, device_ids, status, stopped_by_user, container_id}` for the most recent records |
| `current_models.json` | Which containers are actually `running` vs absent |

**Critical cross-check**: same `board_id` across multiple `device_info` entries = **one physical card, multiple ASICs**. Different `board_id`s = multiple cards. The `num_devices` count and `board_type` together determine which `DeviceConfigurations` enum the backend picks.

### 3. backend.log keyword sweep

```bash
grep -nE 'Raw board_type|Detected.* board type|ChipSlotAllocator initialized|Allocation failed|Manual allocation|Invalid device_id|is_compatible|compatible=False, boards' <bundle>/backend.log | head -80
```

Read those lines. They reveal:
- What `board_type` the backend cached (and the slot count it chose)
- Which models were marked incompatible and why (the boards list)
- The exact device_id rejections with the valid range (`Must be 0-N`)
- Which model the user actually got to deploy

### 4. Failing-deploy logs

For each FAILED deploy in `deployments.json`, read the matching `model_run_logs/model_run_<ts>_<model>_<device>_server.log` to capture the **exact `run.py` command line** (model, workflow, device, device-id flags). This is the request the backend produced.

If a container log exists under `inference_artifacts/docker_server/`, grep it for `Traceback`, `Error`, `OOM`, `Failed`, `tt::` and read context around the first hit. **Pick the right file** by `model_type` from `current_models.json:deploy_cache_entries[].model_impl.model_type`:

- `vllm_<ts>_<model>_<device>_server.log` — LLMs served by vLLM (model_type empty/llm/generation). Traceback patterns: `EngineCore failed to start`, `_set_hf_params`, `AutoConfig.from_pretrained`.
- `media_<ts>_<model>_<device>_server.log` — media-inference-server for TTS (`speecht5_tts`), ASR/STT (`distil-large-v3`, whisper variants). model_type is `tts` or `speech_recognition`. Traceback patterns: container-side `tt::` device errors, model-spec load failures, audio pipeline errors. The Voice Agent pipeline depends on **both** an LLM (vllm log) AND two media services (two media logs) — if the user reports a Voice Agent failure, check all three.

Run logs (`inference_artifacts/run_logs/run_<ts>_id_<impl>_<model>_<device>_server_<hash>.log`) wrap both flavors and contain the `docker run` invocation (mounts, device passthrough, volume name) regardless of whether it's vLLM or media-server.

**Triage gate before fan-out**: if the container log (vllm_*** or media_***) shows `Fabric initialized on N devices` / device init succeeded, and the crash happens *after* that (inside model loading — `AutoConfig.from_pretrained`, `_set_hf_params`, missing tokenizer files, missing audio model files, bad weight shapes), the bug is **not** in the backend allocator / detection / compat code. It's a **model-asset problem inside the persistent docker volume** (`volume_id_<impl>-<model>`). Skip step 5 entirely and jump to the remediation playbook — the chain is hardware ✅ → allocation ✅ → container launch ✅ → model files ❌. Applies to both vLLM and media-server containers.

### 5. Fan out — parallel Explore subagents

**Do this in a single message with 3–4 parallel `Agent` calls** using `subagent_type: Explore`. Brief each one with:
- The specific log evidence (paste the exact lines)
- The specific source files they should read
- A demand for file:line citations and quoted code

Default fan-out targets (adjust per symptom):

| Subagent | Reads | Answers |
|---|---|---|
| **board-type detection** | `app/backend/board_control/services.py` | How does raw tt-smi `board_type` + `num_devices` map to a `DeviceConfigurations`? Is the path through `get_board_type()` or `_extract_board_type_from_data()`? |
| **slot allocator** | `app/backend/docker_control/chip_allocator.py` | How is `total_slots` derived? What's in `MULTI_CHIP_BOARD_SLOTS`? Where does the `Invalid device_id N. Must be 0-M` message come from? |
| **device enum + model compat** | `app/backend/shared_config/device_config.py`, `app/backend/shared_config/model_config.py`, `app/backend/shared_config/models_from_inference_server.json` | All `DeviceConfigurations` values; per-model `device_configurations` lists. Is the user's board in the compat list? |
| **request flow** | `app/backend/docker_control/views.py`, `app/frontend/src/components/<Solution>.tsx`, `app/frontend/src/utils/` | What endpoint received the deploy? What device_ids did the frontend send and why? Is there per-board branching? |

### 6. Synthesize the chain

Always express the result as a **chain**, with each link cited:

```
1. tt-smi reports: <facts from tt_smi.json>
2. Backend normalizes to: <enum> (services.py:<line>)
3. Slot count chosen: <N> (chip_allocator.py:<line>)
4. Frontend requested: device_ids=<list> (<component>.tsx:<line>)
5. Backend rejected: <error> (chip_allocator.py:<line>)
6. User saw: <UI symptom from screenshot / bug report>
```

This makes the broken link obvious.

### 7. Report

End with:
- **Root cause** in one sentence
- **Smallest fix** — the single config/dict change or branch that would unblock, with file:line
- **What it doesn't fix** — call out unresolved UX gaps (e.g. "Voice Agent still needs 3 chips; a 2-ASIC P300 can't run the full pipeline even with the slot fix")

Don't propose patches unless the user asks. Stop after the report.

## Common failure signatures

| Symptom in logs | Likely cause | Source to read |
|---|---|---|
| `ChipSlotAllocator initialized: board=P300c, slots=1` + `Invalid device_id 1` | Slot table missing entry for the detected board enum | `chip_allocator.py` `MULTI_CHIP_BOARD_SLOTS` |
| `compatible=False, boards=[...]` for the requested model | Model's `device_configurations` doesn't include the detected board | `models_from_inference_server.json` |
| `Raw board_type: '<x>'` with unexpected enum chosen | Detection logic miscounts cards vs ASICs | `board_control/services.py` `get_board_type()` |
| Frontend sends device_ids the backend can't honor | Per-board branching missing in the solution UI | `app/frontend/src/components/...SolutionStep.tsx`, `utils/*Placement.ts` |
| `Job <id>` exits without inference log | run.py crashed before docker compose; check model_run_logs `<ts>_<model>_<device>_server.log` for the run.py command and any traceback before it would have emitted | `inference-api/api.py` |
| `ValueError: Unrecognized model in /home/container_app_user/cache_root/model_file_symlinks_map/<model>. Should have a 'model_type' key in its config.json` + `RuntimeError: Engine core initialization failed` | **Persistent docker volume for that model is corrupt / setup never finished** — the HF symlink view is missing `config.json` (or it lacks `model_type`). Tell-tale signs in `deployments.json`: multiple prior deploys of the same model with `status: stopped, stopped_by_user: false, workflow_log_path: null` — the volume has been broken across sessions. Not a code bug — go to the remediation playbook below. | (none — ops fix) |
| Crash inside `model_config.py:_set_hf_params` / `AutoConfig.from_pretrained` / missing tokenizer file | Same as above — incomplete model setup in the volume | (none — ops fix) |

## Remediation playbook — corrupt model volume

When the failure is a model-asset problem (signatures above), the fix is **not a code change** — delete the volume and redeploy. Tell the user:

1. **Verify** the volume is the problem (cheap, read-only):
   ```bash
   docker run --rm -v volume_id_<impl>-<model>:/cache alpine \
     sh -c 'ls -la /cache/model_file_symlinks_map/<model>/ && cat /cache/model_file_symlinks_map/<model>/config.json 2>&1 | head -20'
   ```
   Expect either `No such file or directory` for `config.json`, or a file missing the `"model_type"` key. The volume name format is `volume_id_<impl-with-underscores>-<model-name>` — find it in the `--volume …` line of the failing deploy's run.py log under `inference_artifacts/run_logs/`.

2. **Stop the dead container** (if still present in `current_models.json` / `docker ps -a`):
   ```bash
   docker rm -f <container_name>
   ```

3. **Delete the volume** so `run.py` will re-fetch and re-symlink the model on next deploy:
   ```bash
   docker volume rm volume_id_<impl>-<model>
   ```

4. **Redeploy from the TT-Studio UI.** Confirm `HF_TOKEN` is set in `.env` (the run.py log will say `HF_TOKEN is set`); if not, gated weights won't download.

Call out as an unresolved UX gap in the report: the backend currently polls a dead container for hours emitting `Health check failed: 405: Model is not ready` (`/backend/model_control/model_utils.py:176`) without surfacing the vLLM `Engine core initialization failed` traceback to the user — so this failure looks like an endless spinner. Worth flagging for a separate issue.

## Anti-patterns

- **Don't** start reading source code before you've read the JSON snapshots and grep'd `backend.log` — the logs tell you which code path to read.
- **Don't** run the subagents sequentially. They're independent — single message, multiple `Agent` blocks.
- **Don't** speculate about hardware state from screenshots alone; always ground claims in `tt_smi.json`.
- **Don't** write a fix patch unless asked. The user usually wants the diagnosis first.
- **Don't** assume every failure is a backend/allocator bug. If `Fabric initialized` succeeded and the crash is inside HF model loading, it's a volume/asset problem — recommend `docker volume rm` + redeploy, not source code reading.