# Aie Hw Bringup > Guide to bringing up a new IRON design on real AIE/NPU hardware for the first time — sequential block-by-block (and, for large models, dispatch-by-dispatch) bring-up against a reference, immediate output comparison against the numeric oracle/ONNX Runtime, methodical bisection when something hangs or mismatches, memory-budget tile splits, and validating the host-orchestration handoff between dispatches as its own step. Use this whenever the user is getting a new design running on real NPU hardware for the first time, debugging a hang/deadlock/wrong-output on real hardware (as opposed to in simulation), comparing NPU output against a reference to find where a design diverges, hitting an L1/tile memory budget overflow, or debugging a multi-dispatch handoff — even if they haven't framed it as "bring-up." - Skill: `xilinx/aie-hw-bringup` (Agent Skill) - Install (CLI): `npx skillmds@latest add xilinx/aie-hw-bringup` - Raw SKILL.md: https://api.skillmd.com/api/skills/xilinx/aie-hw-bringup/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Finance & Business - License: Apache-2.0 WITH LLVM-exception - Author: xilinx (https://skillmd.com/u/xilinx) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/xilinx/aie-hw-bringup --- # Hardware bring-up against a reference (the "minimal functional version" phase) Third of four porting phases (see `aie-model-baseline` for the oracle this phase compares against, `aie-dataflow-presim` for the simulation that should already have caught topology/deadlock bugs before this phase starts, `aie-kernel-opt`/`aie-dataflow-opt` for optimization afterward). Simulation catches deadlocks and math bugs cheaply, but it can't catch everything — real hardware has memory budgets, real DMA timing, and real toolchain behavior that a mock doesn't model. This phase is about finding *those* bugs efficiently once you're on real silicon, without re-deriving the debugging process from scratch each time. ## 1. Bring up sequentially — never wire the whole design first Get the smallest possible piece running and bit-exact on hardware first (one block, one layer), then extend by one block/link at a time. At every step, exactly one thing is new since the last known-good state, which is what makes the next section possible. Resist the temptation to wire the full chain because "the simulator already validated it" — the simulator validates the topology and the math; it doesn't validate real DMA timing, real memory layout, or toolchain codegen, so hardware can still fail in ways simulation couldn't predict. If the model spans multiple dispatches (see `aie-model-baseline`), bring up and validate each dispatch on its own first, exactly like a block or layer — then bring up the host-orchestration path that stitches dispatches together as its **own**, separate step. A correct dispatch A and a correct dispatch B do not imply a correct handoff between them: the host code that writes A's output, reloads weights, and invokes B is new surface area that doesn't exist in a single-dispatch design, and it fails in its own ways — a race between when the host reads A's output and when B starts, a layout or scale mismatch between what A wrote and what B expects to read, a weight-reload that didn't happen because a cache thought nothing changed. Treat it as a bring-up target in its own right, not an assumed-correct implementation detail. ## 2. Compare against the reference immediately, on the same input At every step, run the same canary input through both the NPU design and the phase-1 oracle (or ONNX Runtime / the original framework), and diff the output immediately — don't accumulate several new blocks before checking. A single fixed canary input is enough for fast iteration; save broader validation (many inputs, accuracy metrics) for once the design is stable. ## 3. Bisect methodically — one hypothesis, one change, at a time When something hangs or the output mismatches, treat every candidate cause as an independent hypothesis and test it in isolation rather than changing several things and hoping: disable streaming, lower an ObjectFifo depth, split a kernel across two tiles, remove one dependency — one change per test, narrowing to the minimal trigger. If a sibling block/layer already hit a similar-looking failure, try its known fix first before reaching for something novel — a new hypothesis is only worth it once the boring ones are ruled out. **If comparing two versions of a design that behave differently**, a structural diff of the generated MLIR (FIFOs, locks, links, call arguments) between the working and broken version is often faster than staring at either one in isolation — the diff itself points at what changed. Concrete, previously-hit gotchas worth checking early in a bisection: - **Devicename must match the physical hardware.** An npu1 xclbin run on a Strix (npu2) board can return all-zero output silently rather than erroring — check `xrt-smi examine` against the target string before chasing a "wrong output" bug that's actually a device mismatch. - **Stale build artifacts masquerade as a code bug.** Incremental builds track file mtimes, not semantic changes; a `.prj/` directory or a stale `.exe` can silently keep running yesterday's design. When a fix doesn't change behavior, confirm the artifact actually rebuilt before concluding the fix didn't work. - **NPU JIT/xclbin caching can serve a stale binary across runs.** Clear the cache directory between configurations you're trying to compare, and verify with a fresh cache location, not an assumption that "it must have rebuilt." - **Device driver state can persist across runs in ways that look like a hardware fault.** Autosuspend timing and memory fragmentation after repeated rebuild/reload cycles have both produced failures that looked like a design bug but were resolved by a driver-level reset — worth ruling out before assuming the design itself regressed. - **For a multi-dispatch design, check the boundary before the dispatches themselves.** If dispatch A and dispatch B each validate correctly in isolation but the chained result is wrong, suspect the handoff first — a scale/layout mismatch between what A wrote and what B expects, or a stale weight buffer that wasn't reloaded — before re-debugging either dispatch's internals. ## 4. Tile memory budget overflow: split, don't squeeze When a tile's L1 (or a DMA descriptor's budget) overflows because a kernel needs to hold more simultaneous state than fits — multiple activations alive at once for a wide skip/concat, say — the fix that scales is splitting the work across two tiles and cascading the partial result, not squeezing the existing kernel by shrinking precision or unrolling less. Precision/unroll cuts trade away exactly the correctness or performance this phase is trying to establish; a tile split is a dataflow decision that doesn't cost either. ## Checklist before moving to optimization (`aie-kernel-opt` / `aie-dataflow-opt`) - [ ] Every block/layer brought up and validated bit-exact on real hardware, incrementally, not all at once - [ ] Full chain validated bit-exact against the oracle/reference on at least one real canary input - [ ] If the model spans multiple dispatches: each dispatch validated independently, and the host-orchestration handoff between them validated as its own step, not assumed correct because both sides passed in isolation - [ ] Any memory-budget overflow was resolved via a tile split, not a precision/unroll compromise - [ ] Design rules learned during bring-up (minimum FIFO depth for a pattern, a tile split that was necessary) are written down for reuse on the next variant, not left to be rediscovered