tt-studio-debug-bundle

name

tt-studio-debug-bundle

description

Triage a TT-Studio bug-report ZIP / log bundle (named `tt-studio-logs-ttbr-*` or referenced as `ttbr-*`) to root-cause why a model deploy, board allocation, or Voice Agent pipeline failed. Use when the user shares a path to a TT-Studio log bundle, mentions a bug report ID like `ttbr-<hex>`, or attaches a screenshot of the TT-Studio Bug Report issue template. Fans out parallel Explore subagents across `app/backend/board_control/`, `app/backend/docker_control/`, `app/backend/shared_config/`, and `app/frontend/src/components/` to map log evidence to source code.

TT-Studio Debug-Bundle Triage

When this fires

User shares any of:

• A path like /.../tt-studio-logs-ttbr-<hex>/ or a ZIP with the same name
• A ttbr-<hex> reference (the diagnostic ID auto-generated by the TT-Studio Bug Reporter)
• A screenshot of GitHub issue from tenstorrent/tt-studio whose title contains "TT-Studio bug report"
• Log lines like Allocation failed for ...: Invalid device_id, ChipSlotAllocator initialized: board=, Raw board_type:

Bundle layout (assume this shape)

tt-studio-logs-ttbr-<hex>/
├── agent.log                       # AI-agent service (LangGraph, voice, etc.)
├── backend.log                     # Django backend — the main signal
├── current_models.json             # Live docker containers + deploy_cache snapshot
├── deployments.json                # Persisted deploy records (id, device, device_ids, status)
├── docker-control-service.log      # docker daemon control
├── model_run.log                   # FastAPI inference-control server
├── model_run_logs/                 # Per-deploy model run logs:
│   └── model_run_<ts>_<model>_<device>_server.log
├── inference_artifacts/
│   ├── docker_server/              # Container stdout per deploy. Two flavors:
│   │   ├── vllm_<ts>_<model>_<device>_server.log    # LLMs (Llama, Qwen, …)
│   │   └── media_<ts>_<model>_<device>_server.log   # TTS / ASR / Whisper
│   │                               #   (speecht5_tts, distil-large-v3, …)
│   └── run_logs/                   # tt-inference-server run.py logs (one per deploy)
├── startup.log
└── tt_smi.json                     # Raw hardware enumeration from tt-smi

Workflow

Copy this checklist and tick as you go:

Triage Progress:
- [ ] 1. Inventory the bundle
- [ ] 2. Read the three metadata JSONs
- [ ] 3. Pull the smoking-gun lines from backend.log
- [ ] 4. Read the failing deploy's fastapi + inference logs
- [ ] 5. Fan out parallel Explore subagents on source
- [ ] 6. Synthesize the chain (hardware → detection → allocation → request → user-visible error)
- [ ] 7. Report root cause + minimal fix shape

1. Inventory

ls -la <bundle>/ <bundle>/model_run_logs <bundle>/inference_artifacts/docker_server <bundle>/inference_artifacts/run_logs

Note: which models have logs, which deploys are most recent, whether current_models.json shows live containers or an empty deploy_cache_entries.

2. Metadata JSONs — what to extract

File	Pull these fields
tt_smi.json	device_info[].board_info.board_type, bus_id, board_id, ASIC_LOCATION; count of device_info entries
deployments.json	records[].{model_name, device, device_id, device_ids, status, stopped_by_user, container_id} for the most recent records
current_models.json	Which containers are actually running vs absent

Critical cross-check: same board_id across multiple device_info entries = one physical card, multiple ASICs. Different board_ids = multiple cards. The num_devices count and board_type together determine which DeviceConfigurations enum the backend picks.

3. backend.log keyword sweep

grep -nE 'Raw board_type|Detected.* board type|ChipSlotAllocator initialized|Allocation failed|Manual allocation|Invalid device_id|is_compatible|compatible=False, boards' <bundle>/backend.log | head -80

Read those lines. They reveal:

• What board_type the backend cached (and the slot count it chose)
• Which models were marked incompatible and why (the boards list)
• The exact device_id rejections with the valid range (Must be 0-N)
• Which model the user actually got to deploy

4. Failing-deploy logs

For each FAILED deploy in deployments.json, read the matching model_run_logs/model_run_<ts>_<model>_<device>_server.log to capture the exact run.py command line (model, workflow, device, device-id flags). This is the request the backend produced.

If a container log exists under inference_artifacts/docker_server/, grep it for Traceback, Error, OOM, Failed, tt:: and read context around the first hit. Pick the right file by model_type from current_models.json:deploy_cache_entries[].model_impl.model_type:

• vllm_<ts>_<model>_<device>_server.log — LLMs served by vLLM (model_type empty/llm/generation). Traceback patterns: EngineCore failed to start, _set_hf_params, AutoConfig.from_pretrained.
• media_<ts>_<model>_<device>_server.log — media-inference-server for TTS (speecht5_tts), ASR/STT (distil-large-v3, whisper variants). model_type is tts or speech_recognition. Traceback patterns: container-side tt:: device errors, model-spec load failures, audio pipeline errors. The Voice Agent pipeline depends on both an LLM (vllm log) AND two media services (two media logs) — if the user reports a Voice Agent failure, check all three.

Run logs (inference_artifacts/run_logs/run_<ts>_id_<impl>_<model>_<device>_server_<hash>.log) wrap both flavors and contain the docker run invocation (mounts, device passthrough, volume name) regardless of whether it's vLLM or media-server.

Triage gate before fan-out: if the container log (vllm_*** or media_***) shows Fabric initialized on N devices / device init succeeded, and the crash happens after that (inside model loading — AutoConfig.from_pretrained, _set_hf_params, missing tokenizer files, missing audio model files, bad weight shapes), the bug is not in the backend allocator / detection / compat code. It's a model-asset problem inside the persistent docker volume (volume_id_<impl>-<model>). Skip step 5 entirely and jump to the remediation playbook — the chain is hardware ✅ → allocation ✅ → container launch ✅ → model files ❌. Applies to both vLLM and media-server containers.

5. Fan out — parallel Explore subagents

Do this in a single message with 3–4 parallel Agent calls using subagent_type: Explore. Brief each one with:

• The specific log evidence (paste the exact lines)
• The specific source files they should read
• A demand for file:line citations and quoted code

Default fan-out targets (adjust per symptom):

Subagent	Reads	Answers
board-type detection	app/backend/board_control/services.py	How does raw tt-smi board_type + num_devices map to a DeviceConfigurations? Is the path through get_board_type() or _extract_board_type_from_data()?
slot allocator	app/backend/docker_control/chip_allocator.py	How is total_slots derived? What's in MULTI_CHIP_BOARD_SLOTS? Where does the Invalid device_id N. Must be 0-M message come from?
device enum + model compat	app/backend/shared_config/device_config.py, app/backend/shared_config/model_config.py, app/backend/shared_config/models_from_inference_server.json	All DeviceConfigurations values; per-model device_configurations lists. Is the user's board in the compat list?
request flow	app/backend/docker_control/views.py, app/frontend/src/components/<Solution>.tsx, app/frontend/src/utils/	What endpoint received the deploy? What device_ids did the frontend send and why? Is there per-board branching?

6. Synthesize the chain

Always express the result as a chain, with each link cited:

1. tt-smi reports: <facts from tt_smi.json>
2. Backend normalizes to: <enum> (services.py:<line>)
3. Slot count chosen: <N> (chip_allocator.py:<line>)
4. Frontend requested: device_ids=<list> (<component>.tsx:<line>)
5. Backend rejected: <error> (chip_allocator.py:<line>)
6. User saw: <UI symptom from screenshot / bug report>

This makes the broken link obvious.

7. Report

End with:

• Root cause in one sentence
• Smallest fix — the single config/dict change or branch that would unblock, with file:line
• What it doesn't fix — call out unresolved UX gaps (e.g. "Voice Agent still needs 3 chips; a 2-ASIC P300 can't run the full pipeline even with the slot fix")

Don't propose patches unless the user asks. Stop after the report.

Common failure signatures

Symptom in logs	Likely cause	Source to read
ChipSlotAllocator initialized: board=P300c, slots=1 + Invalid device_id 1	Slot table missing entry for the detected board enum	chip_allocator.py MULTI_CHIP_BOARD_SLOTS
compatible=False, boards=[...] for the requested model	Model's device_configurations doesn't include the detected board	models_from_inference_server.json
Raw board_type: '<x>' with unexpected enum chosen	Detection logic miscounts cards vs ASICs	board_control/services.py get_board_type()
Frontend sends device_ids the backend can't honor	Per-board branching missing in the solution UI	app/frontend/src/components/...SolutionStep.tsx, utils/*Placement.ts
Job <id> exits without inference log	run.py crashed before docker compose; check model_run_logs <ts>_<model>_<device>_server.log for the run.py command and any traceback before it would have emitted	inference-api/api.py
ValueError: Unrecognized model in /home/container_app_user/cache_root/model_file_symlinks_map/<model>. Should have a 'model_type' key in its config.json + RuntimeError: Engine core initialization failed	Persistent docker volume for that model is corrupt / setup never finished — the HF symlink view is missing config.json (or it lacks model_type). Tell-tale signs in deployments.json: multiple prior deploys of the same model with status: stopped, stopped_by_user: false, workflow_log_path: null — the volume has been broken across sessions. Not a code bug — go to the remediation playbook below.	(none — ops fix)
Crash inside model_config.py:_set_hf_params / AutoConfig.from_pretrained / missing tokenizer file	Same as above — incomplete model setup in the volume	(none — ops fix)

Remediation playbook — corrupt model volume

When the failure is a model-asset problem (signatures above), the fix is not a code change — delete the volume and redeploy. Tell the user:

1. Verify the volume is the problem (cheap, read-only):

   docker run --rm -v volume_id_<impl>-<model>:/cache alpine \
     sh -c 'ls -la /cache/model_file_symlinks_map/<model>/ && cat /cache/model_file_symlinks_map/<model>/config.json 2>&1 | head -20'
   

   Expect either No such file or directory for config.json, or a file missing the "model_type" key. The volume name format is volume_id_<impl-with-underscores>-<model-name> — find it in the --volume … line of the failing deploy's run.py log under inference_artifacts/run_logs/.

2. Stop the dead container (if still present in current_models.json / docker ps -a):

   docker rm -f <container_name>
   

3. Delete the volume so run.py will re-fetch and re-symlink the model on next deploy:

   docker volume rm volume_id_<impl>-<model>
   

4. Redeploy from the TT-Studio UI. Confirm HF_TOKEN is set in .env (the run.py log will say HF_TOKEN is set); if not, gated weights won't download.

Call out as an unresolved UX gap in the report: the backend currently polls a dead container for hours emitting Health check failed: 405: Model is not ready (/backend/model_control/model_utils.py:176) without surfacing the vLLM Engine core initialization failed traceback to the user — so this failure looks like an endless spinner. Worth flagging for a separate issue.

Anti-patterns

• Don't start reading source code before you've read the JSON snapshots and grep'd backend.log — the logs tell you which code path to read.
• Don't run the subagents sequentially. They're independent — single message, multiple Agent blocks.
• Don't speculate about hardware state from screenshots alone; always ground claims in tt_smi.json.
• Don't write a fix patch unless asked. The user usually wants the diagnosis first.
• Don't assume every failure is a backend/allocator bug. If Fabric initialized succeeded and the crash is inside HF model loading, it's a volume/asset problem — recommend docker volume rm + redeploy, not source code reading.