# Triage Dtype Bfloat16 > Triage one tt-forge-models training test failing with a bfloat16 dtype-mismatch RuntimeError (e.g. "mat1 and mat2 must have the same dtype, but got Float and BFloat16", "'' not implemented for 'BFloat16'"). For cross-dtype operands, attempts a minimal loader fix propagating `dtype_override` into the offending tensor constructor, then re-runs CPU + pytest and updates the YAML (passing -> EXPECTED_PASSING; new failure -> KNOWN_FAILURE_XFAIL). For op-not-implemented (no PyTorch kernel), goes straight to KNOWN_FAILURE_XFAIL with the verbatim error. Updates every training entry sharing the affected loader. Never edits inference YAML or `dynamic_loader.py`. - Skill: `tenstorrent/triage-dtype-bfloat16` (Agent Skill) - Install (CLI): `npx skillmds@latest add tenstorrent/triage-dtype-bfloat16` - Raw SKILL.md: https://api.skillmd.com/api/skills/tenstorrent/triage-dtype-bfloat16/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: tenstorrent (https://skillmd.com/u/tenstorrent) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tenstorrent/triage-dtype-bfloat16 --- # Triage bfloat16 dtype-mismatch training failures This skill handles **one** failure pattern only: a training-mode `RuntimeError` whose text contains `BFloat16` plus a dtype-mismatch or "not implemented for BFloat16" signature. Examples: - `RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16` - `RuntimeError: 'deformable_im2col' not implemented for 'BFloat16'` - `RuntimeError: Index put requires the source and destination dtypes match, got Float for the destination and BFloat16 for the source.` - `RuntimeError: expected scalar type Float but found BFloat16` (and similar `expected … got BFloat16` shapes) It works on exactly one test (passed as ``). If the failure reason is anything else, abort immediately. The skill recognizes **two flavors** of bfloat16 failure and treats them differently: - **Cross-dtype operands** (e.g. `mat1 and mat2 must have the same dtype, but got Float and BFloat16`, `Index put … dtypes match`, `expected scalar type Float but found BFloat16`). Almost always caused by a tensor that the loader constructs without honoring `dtype_override` - `torch.zeros(...)`, `torch.ones(...)`, `torch.tensor(...)`, etc. without a `dtype=` argument. The skill **attempts a minimal loader fix** that adds `dtype=dtype_override` to the offending constructor, then re-verifies. If the fix works the test may now pass (mark `EXPECTED_PASSING`) or progress to a different unrelated failure (mark `KNOWN_FAILURE_XFAIL` with the new reason). If the fix doesn't apply (the offending tensor isn't in the loader), fall back to the xfail-only path. - **Op-not-implemented for BFloat16** (e.g. `'deformable_im2col' not implemented for 'BFloat16'`). A specific PyTorch op simply lacks a bfloat16 kernel. **Not loader-fixable.** Go straight to `KNOWN_FAILURE_XFAIL` with the verbatim error. ## Argument: `` accepted forms Auto-detect both: - **YAML key** - e.g. `centernet/pytorch-Dla1x_Coco-single_device-training`. Looks like `/---`. - **`model_info.name`** - e.g. `pytorch_CenterNet_Dla1x_Coco_cv_object_det_github`. Composed in `third_party/tt_forge_models/config.py` as `f"{framework}_{model}_{variant}_{task}_{source}"`. ## Background (load into context, do not paraphrase to the user) - The runner forces bfloat16 for both training and inference. `tests/runner/utils/dynamic_loader.py:362-419` unconditionally passes `dtype_override=torch.bfloat16` to `load_model` and `load_inputs` whenever the loader signature accepts it. There is no per-test float32 escape hatch in the YAML or runner. - All loaders in `third_party/tt_forge_models//pytorch/loader.py` default to **float32** when `dtype_override` is not passed. HuggingFace loaders pass `torch_dtype=dtype_override` to `from_pretrained`; non-HF loaders call `.to(dtype_override)` after loading. So a CPU script can flip dtypes simply by passing or omitting that kwarg. - Where the error fires. The CPU forward + backward at `tests/infra/testers/single_chip/model/torch_model_tester.py:244-263` runs in plain PyTorch *before* any TT compilation. `cpu_res = unpack_forward_output(model(**inputs))` is followed by `cpu_res.backward(gradient=...)`, and both bfloat16 op-not-implemented and cross-dtype mismatch errors fire there. Reproducing the failure on plain CPU is therefore conclusive - no hardware involved. - Why training-mode hits this more than inference. Forward in bfloat16 often succeeds; the failing ops are usually backward-only (e.g. autograd of `index_put`, deformable conv backward). For cross-dtype operands the same float32 input may be silently upcast in a forward kernel but trigger a strict dtype check in a different op or in the backward. Inference YAML entries for the same model are often `EXPECTED_PASSING` even when training fails. Note that the runner forces bfloat16 for *inference too*, so a loader-side fix can affect inference behavior - see Step 7's check for stale inference entries. - **Two failure flavors -> two code paths in this skill.** - **Cross-dtype operands** -> try a loader fix in Step 5, then act on the post-fix outcome. - **Op-not-implemented for BFloat16** -> skip the loader-fix step entirely, go to Step 6. - **Loader-wide scope.** When one variant of a model fails on a fixed dtype-mismatch or op-not-implemented in shared loader code, every variant of that loader that exercises the same path fails identically. Loader fixes and YAML updates apply to every training entry sharing the affected loader, not only the one passed in. Variants whose loader path is the same but whose CPU bfloat16 phase actually passes are left untouched. Confirm per-variant before adding to the update set. - Test-runner gates on YAML status: `NOT_SUPPORTED_SKIP` entries are skipped at collection (`tests/runner/test_models.py:115-119`) and `KNOWN_FAILURE_XFAIL` entries are imperatively xfailed via `pytest.xfail(reason)` (`tests/runner/test_utils.py:750`). Verification uses `pytest --force-run --runxfail` to bypass both without editing the YAML: `--force-run` (`tests/runner/conftest.py:112-117`) makes `NOT_SUPPORTED_SKIP` entries run their body, `--runxfail` neutralizes the imperative xfail so the test reports its real outcome instead of XFAIL. Apply the YAML edit once, after observing the real outcome. - `third_party/tt_forge_models` is a git submodule with its own working tree. Loader edits land there. **Do not commit in the submodule.** Leave changes uncommitted so the user can review. ## Workflow ### Step 1 - Resolve `` -> YAML entry + loader path 1. If `` matches `^[a-z0-9_]+/.+-(single_device|data_parallel|tensor_parallel)-(training|inference)$`, treat as a YAML key. Otherwise treat as a `model_info.name` string. 2. **YAML key path:** locate the entry in `tests/runner/test_config/torch/test_config_training_single_device.yaml`. Loader path is `third_party/tt_forge_models//pytorch/loader.py` where `` is the segment before the first `/` in the key. 3. **`model_info.name` path:** parse as `____`. The `framework` is always `pytorch` for this skill. Search loaders with `grep -rln "model=.*\"\"\|ModelName.*" third_party/tt_forge_models/`, then for each candidate read its `_get_model_info` to confirm variant/task/source. Once the loader is identified, derive the YAML key as `/pytorch--single_device-training` and confirm it exists. 4. If neither resolves, abort with: `Could not resolve to a YAML entry or loader. Provide either the YAML key or the model_info.name.` 5. If `` is a `model_info.name` that resolves to multiple loader candidates, abort and ask the user to provide the YAML key. ### Step 2 - Gate on the failure pattern 1. Read the YAML entry. Its `reason:` must contain `BFloat16` and at least one of: `must have the same dtype`, `not implemented for 'BFloat16'`, `dtypes match`, `expected scalar type`. If not, abort: `This skill only handles bfloat16 dtype-mismatch training failures. The failing reason here is: .` 2. Classify the flavor: - `not implemented for 'BFloat16'` -> **op-not-implemented**, set `flavor = "op_not_implemented"`. - `must have the same dtype`, `dtypes match`, or `expected scalar type` -> **cross-dtype operands**, set `flavor = "cross_dtype"`. 3. Note the prior `status` and `bringup_status` for the report. They may already be `NOT_SUPPORTED_SKIP` / `FAILED_FE_COMPILATION` from a stale triage - that is fine, this skill will migrate them to `KNOWN_FAILURE_XFAIL` (or `EXPECTED_PASSING` after a successful fix) regardless. ### Step 3 - CPU triage script (combined bfloat16 + float32, with input-dtype dump) One script, one process, two phases. Per-phase try/except so a bfloat16 raise doesn't skip the float32 phase. The script also prints the dtype of every tensor returned by `load_inputs(dtype_override=torch.bfloat16)` - that's the diagnostic for Step 5. Before writing the script, check whether the loader defines a `ModelVariant` enum: ```bash grep -n "class ModelVariant" third_party/tt_forge_models//pytorch/loader.py ``` If the grep returns a match, include `, ModelVariant` in the import and use `ModelLoader(variant=ModelVariant.)`. If it returns nothing, drop both - use `ModelLoader()` with no `variant` argument. Write `/tmp/triage_dtype__.py`: ```python import sys, traceback, torch sys.path.insert(0, ".") from third_party.tt_forge_models..pytorch.loader import ModelLoader # add , ModelVariant if used def _dump_input_dtypes(label, inputs): print(f"[{label}] Input tensor dtypes:", flush=True) if isinstance(inputs, dict): items = inputs.items() elif isinstance(inputs, (list, tuple)): items = enumerate(inputs) else: items = [("", inputs)] for k, v in items: if isinstance(v, torch.Tensor): print(f" {k}: dtype={v.dtype} shape={tuple(v.shape)}", flush=True) else: print(f" {k}: not a tensor ({type(v).__name__})", flush=True) def run(label, dtype): print(f"\n===== {label} (dtype_override={dtype}) =====", flush=True) try: loader = ModelLoader(variant=ModelVariant.) # drop variant arg if loader has no ModelVariant kw = {} if dtype is None else {"dtype_override": dtype} model = loader.load_model(**kw) model.train() inputs = loader.load_inputs(**kw) _dump_input_dtypes(label, inputs) out = (model(**inputs) if isinstance(inputs, dict) else model(*inputs) if isinstance(inputs, (list, tuple)) else model(inputs)) unpacked = loader.unpack_forward_output(out) # Mimic torch_model_tester._test_training: random-gradient backward over the unpacked tensor. if isinstance(unpacked, torch.Tensor): grad = torch.randn(unpacked.shape, dtype=unpacked.dtype) unpacked.backward(gradient=grad) print(f"{label}: OK -- forward+backward completed", flush=True) else: print( f"{label}: unpack_forward_output returned {type(unpacked).__name__}, " f"not a single Tensor. Forward succeeded; skipping backward.", flush=True, ) print( f"{label}: Multi-tensor outputs need a custom CPU repro that " f"sums all leaf tensors before .backward(). The forward dtype dump " f"above is still valid for cross-dtype diagnosis.", flush=True, ) except Exception as e: print(f"{label}: FAILED -- {type(e).__name__}: {e}", flush=True) traceback.print_exc() run("BFLOAT16", torch.bfloat16) run("FLOAT32", None) # None = use loader default, which is float32 ``` Run from the tt-xla repo root with `source venv/activate` already done: ```bash python /tmp/triage_dtype__.py &> /tmp/triage_dtype__.log ``` **Read the log with the Read tool. Never use `tail` or `less` - they hide errors behind generic exit codes** (this is a hard rule from feedback memory). If `load_inputs` requires a non-default kwarg (e.g. `seq_len`, `batch_size`), mirror what the runner passes and re-run. To find the exact kwargs the runner uses for this model, grep: ```bash grep -n "" tests/runner/utils/dynamic_loader.py ``` This surfaces any model-specific overrides in the `380-419` block. Use those kwargs verbatim in the script's `kw` dict alongside `dtype_override`. If `loader.unpack_forward_output(...)` itself raises in either phase, the model also has the unrelated `unpack_forward_output` failure. Abort and tell the user to run the `triage-unpack-forward-output` skill first. ### Step 4 - Apply the decision tree Read the two phase headers (`BFLOAT16: …` and `FLOAT32: …`) from the log: | BFLOAT16 phase | FLOAT32 phase | Action | | --- | --- | --- | | FAILED with `BFloat16` + dtype-mismatch / not-implemented | OK | **Proceed.** Capture the bfloat16 `RuntimeError: …` line verbatim. Continue. | | FAILED with `BFloat16` signature | FAILED (any) | **Abort, escalate.** Model has a non-bfloat16 problem too. Print both error excerpts in the report; do not edit anything. | | OK | (any) | **Abort.** Failure didn't reproduce on CPU - YAML entry may be stale or the env differs. Tell the user to re-run pytest first; do not edit anything. | | FAILED but error has no `BFloat16` / dtype-mismatch | (any) | **Abort.** Pattern gate misfired or upstream change moved the failure mode. Tell the user to use a different skill. | If "Error code 13" is the only thing in the log, that is generic - `grep -nE "RuntimeError" /tmp/triage_dtype__.log` and use the first matching line as the canonical error. Branch on `flavor`: - `flavor == "op_not_implemented"` -> **skip Step 5, jump to Step 6** (xfail-only, no loader fix). - `flavor == "cross_dtype"` -> **continue to Step 5** to attempt a loader fix. ### Step 5 - (cross-dtype only) Attempt a loader fix The cross-dtype error is almost always caused by a tensor in `load_inputs` (sometimes `load_model`) that's constructed without honoring `dtype_override`. The dtype dump from Step 3 makes the offender obvious: in the BFLOAT16 phase any *floating-point* input whose dtype is `torch.float32` (instead of `torch.bfloat16`) is a candidate. Integer/bool tensors (`torch.long`, `torch.int64`, `torch.bool`) are expected and should not be touched. 1. **Identify the offending tensor.** Read `[BFLOAT16] Input tensor dtypes:` block from the log. List every entry with `dtype=torch.float32`. If there is exactly one such tensor, that's the offender. If there are several, the bfloat16 traceback often names the operand (e.g., `decoder_input_values`); cross-reference with the input keys. 2. **Find the constructor in the loader.** Open `third_party/tt_forge_models//pytorch/loader.py`. Grep for tensor constructors that don't take `dtype=`: ```bash grep -nE "torch\.(zeros|ones|full|empty|randn|rand|randint|tensor|arange|eye|linspace)\(" third_party/tt_forge_models//pytorch/loader.py ``` For each match, check whether `dtype=` already appears in the call. If not, and the constructor produces the offending input from step 1, that's your fix site. Common pattern: ```python decoder_input_values = torch.zeros((1, 1, num_mel_bins)) # before decoder_input_values = torch.zeros((1, 1, num_mel_bins), dtype=dtype_override) # after ``` `torch.zeros(shape, dtype=None)` falls back to the global default (float32), so passing `dtype=dtype_override` is safe in the unset path too. For tensors that come from external sources (a `.npy` file, a processor that always returns float32, etc.), the fix is `t = t.to(dtype_override) if dtype_override is not None else t` after the load. **Scope of the loader edit.** Allowed: add `dtype=dtype_override` to a tensor constructor, or add a single `.to(dtype_override)` cast on a tensor returned by a non-dtype-aware helper. Forbidden: any other refactoring, renaming, helper extraction, or behavior change. One offender per invocation; if the diagnostic shows several float32 offenders that aren't all clearly traceable to fixable constructors, fall back to xfail (Step 6) and note the situation in the report. 3. **If no fix site is identifiable** (e.g., the input is built by a HuggingFace processor that we can't reach without monkey-patching, or all candidate constructors already pass `dtype=dtype_override`), set `loader_fix_attempted = False` and **jump to Step 6** (xfail-only). 4. **Apply the fix, re-run the CPU triage script.** Read the new log. There are three outcomes: | New CPU outcome | Decision | | --- | --- | | BFLOAT16: OK and FLOAT32: OK | **Loader fix worked at the CPU level.** Set `loader_fix_attempted = True`, `loader_fix_works_cpu = True`. Continue to Step 6 to determine TT-XLA outcome. | | BFLOAT16: still FAILED with the same dtype-mismatch | **Wrong fix site or the bug isn't loader-side.** Revert the loader edit (`git -C third_party/tt_forge_models checkout -- /pytorch/loader.py`). Set `loader_fix_attempted = False`. Continue to Step 6 (xfail-only). | | BFLOAT16: FAILED but with a *different* error (no longer dtype-mismatch) | **Loader fix moved the failure to CPU.** This is rare (e.g. backward of an op that doesn't support bfloat16). Keep the loader edit (it's correct) - the new error becomes the canonical reason. Set `loader_fix_attempted = True`, `loader_fix_works_cpu = False`, capture the new error verbatim, continue to Step 6. | ### Step 6 - Determine loader-level scope Skill operates loader-wide (every training entry pointing to the same `third_party/tt_forge_models//pytorch/loader.py`). Enumerate them: ```bash grep -nE "^ /pytorch.*-single_device-training:" tests/runner/test_config/torch/test_config_training_single_device.yaml ``` For loaders with multiple variants where some pass on TT (`status: EXPECTED_PASSING`) before the fix, do **not** assume those also failed on bfloat16. Re-run the CPU triage script for each candidate variant before adding it to the update set. After a successful loader fix, every previously-failing variant should now have CPU bfloat16 OK; if any still fails, do not include it in the update set and flag it for human review. ### Step 7 - Verify with pytest, then update the YAML Do **not** pre-edit the YAML. Verification runs the tests with `--force-run --runxfail` so the current YAML status is bypassed without an intermediate flip: - `--force-run` makes `NOT_SUPPORTED_SKIP` entries execute (`tests/runner/conftest.py:112-117`, `tests/runner/test_models.py:115-119`). - `--runxfail` makes the runner's imperative `pytest.xfail(reason)` (`tests/runner/test_utils.py:750`) report the real outcome instead of XFAIL - covers stale `KNOWN_FAILURE_XFAIL` entries. Use `--junitxml=` to capture structured per-testcase outcomes - the runner records the detailed error (TT_FATAL / OOM / `error:` / `RuntimeError` lines) into `` via `record_property("error_message", …)` (`tests/runner/test_utils.py:737`), and pytest writes any failure traceback into ``. The XML is the source of truth; also dump stdout/stderr to a `.log` companion in case the XML is ambiguous (e.g. only "Error code 13" with no `error_message`). Run the affected training tests in a single pytest invocation: ```bash timeout 1200 pytest --force-run --runxfail \ --junitxml=/tmp/verify_dtype__bf16_training.xml \ "tests/runner/test_models.py::test_all_models_torch[-single_device-training]" \ "tests/runner/test_models.py::test_all_models_torch[-single_device-training]" \ &> /tmp/verify_dtype__bf16_training.log ``` Read the XML with the Read tool. Use the `.log` companion only if the XML is missing detail. Then write the final YAML once per affected entry based on the JUnit outcome. #### Branch A — op-not-implemented, or cross-dtype without a successful loader fix No loader change. The pytest verification above confirms the entry still hits the same CPU bfloat16 error before TT compilation: with `--runxfail` the imperative xfail is neutralized, so the JUnit XML shows `` whose `error_message` matches the `RuntimeError: …BFloat16…` captured in Step 4. Then write the YAML once with the canonical bfloat16 reason from the table below: ```yaml /pytorch--single_device-training: status: KNOWN_FAILURE_XFAIL bringup_status: FAILED_FE_COMPILATION reason: "" ``` #### Branch B — cross-dtype, loader fix succeeded on CPU Loader fix succeeded at the CPU level. The pytest verification above runs against the existing YAML state (no flip needed, thanks to `--force-run --runxfail`); read the JUnit XML to decide the final YAML state per entry. For each ``: - **No `` / `` / `` child** -> test now passes on TT-XLA. Write: ```yaml /pytorch--single_device-training: status: EXPECTED_PASSING ``` Drop `bringup_status` and `reason`. `EXPECTED_PASSING` is only set after a clean pytest pass, not based on CPU evidence alone. - **`` whose `error_message` is a PCC / atol / numerics divergence** (e.g. `PCC = …`, `atol`, `Tensor mismatch`): the test is functionally running, just inaccurate. Write: ```yaml /pytorch--single_device-training: status: EXPECTED_PASSING assert_pcc: false ``` Drop `bringup_status` and `reason`. The runner consumes `assert_pcc: false` (`tests/runner/test_utils.py:154,220-223`) to disable the PCC check; precedent: `qwen_1_5/causal_lm/pytorch-0.5B-single_device-training`. - **`` whose `error_message` is a *new, non-bfloat16* error** (`error:` / `RuntimeError` / `TT_FATAL` / MLIR pass failure, but **not** a PCC/numerics divergence): the test now fails for an unrelated TT-XLA / TT-MLIR reason. Write the final YAML with the new error verbatim: ```yaml /pytorch--single_device-training: status: KNOWN_FAILURE_XFAIL bringup_status: FAILED_TTMLIR_COMPILATION # or FAILED_FE_COMPILATION / FAILED_RUNTIME - see bringup_status selection table below reason: "" ``` Use the `error_message` property as the canonical one-liner. If it's missing or generic ("Error code 13"), fall back to the `.log`: `grep -nE "TT_FATAL|TT_THROW|error:|RuntimeError" /tmp/verify_dtype__bf16_training.log` and use the first matching line. Skip noise like deprecation warnings and "Found an argument on non-XLA device". - **`` whose `error_message` is still the original bfloat16 dtype-mismatch**: the loader fix passed CPU but something else (the runner re-creates inputs, or a buffer is constructed inside the model when not on CPU) reintroduces the float32 tensor. Revert the loader fix and fall back to Branch A. Flag for human review. - **`` or ``** -> unexpected with `--force-run --runxfail`; means a different skip path triggered. Flag for human review. #### Branch C — cross-dtype, loader fix applied but new CPU error appeared Loader fix is correct (it eliminated the dtype-mismatch) but a different CPU error appeared. Keep the loader edit. YAML stays `KNOWN_FAILURE_XFAIL` but with the new CPU error as the reason: ```yaml /pytorch--single_device-training: status: KNOWN_FAILURE_XFAIL bringup_status: FAILED_FE_COMPILATION reason: "" ``` The pytest verification above (with `--force-run --runxfail`) confirms the entry actually fails - the JUnit XML must show `` whose `error_message` matches the new CPU error, not unexpected `` with no children (which would mean an `XPASS` and require human review). #### `bringup_status` selection Three stages exist (defined in `tests/utils.py:BringupStatus`): | Situation | Use | | --- | --- | | CPU forward/backward errors out before TT compilation is ever invoked - bfloat16 op errors, cross-dtype mismatches, Python exceptions in the loader | `FAILED_FE_COMPILATION` | | TT-MLIR's compilation pipeline fails - MLIR lowering passes, dialect errors (e.g. `error: 'ttir.*'`), `ElementsAttr` assertion, process abort (SIGABRT / exit 134) from the compiler | `FAILED_TTMLIR_COMPILATION` | | Compilation succeeds but the test fails during device kernel execution | `FAILED_RUNTIME` | | Numerics divergence (PCC / atol failure) after a loader fix | not a `bringup_status` - set `status: EXPECTED_PASSING` with `assert_pcc: false` instead | | Test now passes on TT-XLA | drop `bringup_status` (status: EXPECTED_PASSING is enough) | **How to classify a new error from pytest verification (Branch B):** look at the `error_message` and failure traceback in the JUnit XML: - Contains `error:`, `ttir.`, `ttnn.`, `ElementsAttr`, `Aborted`, or `core dumped` -> `FAILED_TTMLIR_COMPILATION` - Contains `TT_FATAL` / `TT_THROW` with kernel/dispatch/device context -> `FAILED_RUNTIME` - Python exception (`RuntimeError`, `ValueError`, etc.) before any TT call -> `FAILED_FE_COMPILATION` #### Canonical bfloat16 reason strings (Branch A) `reason` - verbatim, single-line, double-quoted. Match these canonical strings exactly when they apply: | CPU bfloat16 error | Canonical `reason` | | --- | --- | | `RuntimeError: '' not implemented for 'BFloat16'` | `"RuntimeError: '' not implemented for 'BFloat16'"` | | `RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16` | `"RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16"` | | `RuntimeError: Index put requires the source and destination dtypes match, got Float for the destination and BFloat16 for the source.` | `"RuntimeError: Index put requires the source and destination dtypes match, got Float for the destination and BFloat16 for the source."` | | `RuntimeError: expected scalar type Float but found BFloat16` | `"RuntimeError: expected scalar type Float but found BFloat16"` | Trim only if the error line is longer than ~120 chars; never re-paraphrase. Do not re-wrap or re-punctuate. #### Inference YAML Do not edit the inference YAML by default. **One exception:** if a loader fix landed (Branch B/C) AND the inference YAML has a stale entry (`KNOWN_FAILURE_XFAIL` with a bfloat16 dtype-mismatch reason for the same loader), call this out in the final report so the user can re-triage inference manually. Don't auto-edit inference - inference and training can fail at different points and the diagnostic for inference is out of scope here. ### Step 8 - Final report Print a single concise summary to the user: 1. Resolved test -> loader path + YAML key. 2. Failure flavor (`op_not_implemented` or `cross_dtype`). 3. CPU outcomes pre-fix - `BFLOAT16: FAILED - `, `FLOAT32: OK` (or whichever decision-tree row applied). 4. Loader fix attempted? If yes, the one-line edit (file + line) and the post-fix CPU outcome. 5. Loader-wide scope - list of all training entries updated. 6. Per-entry post-fix outcome and YAML state set: `EXPECTED_PASSING`, `KNOWN_FAILURE_XFAIL` (with which reason), or unchanged. 7. Pytest verification outcome per entry (or note that dataset gating prevented verification). 8. Sanity-grep confirming the inference YAML was not touched: `git diff tests/runner/test_config/torch/test_config_inference_single_device.yaml` should be empty (or note any flagged stale inference entries). 9. Anything that needs human review (e.g. float32 also failed, model also has an `unpack_forward_output` issue, an unexpected variant of the same loader needed to be excluded, inference entry is stale). 10. Submodule reminder if a loader edit was applied: `third_party/tt_forge_models` is a submodule - the loader change is uncommitted in its working tree; the user may want to commit it on a submodule branch and bump the parent pointer. ## Hard rules (do not violate) - One test at a time. If `` is missing or matches multiple loader candidates, abort and ask. - Only act when `reason:` contains `BFloat16` plus one of `must have the same dtype`, `not implemented for 'BFloat16'`, `dtypes match`, or `expected scalar type … BFloat16`. Anything else: abort, do not edit. - Both CPU phases (bfloat16 and float32) must run. If bfloat16 fails on CPU but float32 also fails, abort and escalate to the user - that is a different problem class. Do not edit YAML or loader. - For `op_not_implemented` flavor: never edit the loader. The PyTorch op simply lacks a bfloat16 kernel. - For `cross_dtype` flavor: loader edits are limited to **adding `dtype=dtype_override` to a single tensor constructor** in `load_inputs` / `load_model`, or **adding one `.to(dtype_override)` cast** on a non-dtype-aware tensor returned by a helper. No other refactoring, renaming, or behavior change. One offender per invocation. If multiple unfixable offenders exist or the offender isn't in the loader, fall back to xfail. - Do not modify `python_package/tt_torch/torch_overrides.py` or `tests/runner/utils/dynamic_loader.py`. Per-test triage; global fixes are out of scope. - Do not edit `tests/runner/test_config/torch/test_config_inference_single_device.yaml`. If a loader fix may have impacted inference, flag the relevant inference entries in the final report for human review. - Loader-wide YAML update: every training entry pointing to the same loader file must be updated, not only the one passed in - but only after confirming per-variant CPU reproduction. - Reason string is verbatim. Match the canonical forms in the Step 7 table exactly for `op_not_implemented`. For new errors after a loader fix, copy the most signal-bearing error line verbatim; do not paraphrase. - Use `status: KNOWN_FAILURE_XFAIL` (not `NOT_SUPPORTED_SKIP`). Tests stay live as xfails so we notice when bfloat16 support arrives upstream. Only set `EXPECTED_PASSING` after a clean pytest pass on TT-XLA, or after pytest reports a PCC-only divergence (in which case `status: EXPECTED_PASSING` with `assert_pcc: false`) - never based on CPU evidence alone. - Verification uses `pytest --force-run --runxfail --junitxml=`. Do not pre-edit the YAML before pytest. Read the XML for outcomes; classify failures from the `` element and the `error_message` property. - When migrating a stale `NOT_SUPPORTED_SKIP` entry, edit it in place to the post-fix target status (`EXPECTED_PASSING` if pytest passes, `KNOWN_FAILURE_XFAIL` otherwise). - Never use `tail` or `less` on pytest output. Dump to a file with `&>`, then Read. - "Error code 13" is generic - always look further for the real `RuntimeError` line. - Do not commit or push. Leave changes staged-but-uncommitted unless the user asks otherwise. `third_party/tt_forge_models` is a submodule with its own git tree - also do not commit there; let the user handle the submodule pointer bump.