# Aiesimloaddebug > --- name: aiesimloaddebug description: Debug AIE simulator (aie2pssimmsm) segfaults. Two families — (1) load-time crash at PS.so load, before "AIEHLC PS IP started", usually a stale script/sim/build/kernel_elf_init.cc referencing the wrong _binary_kernel__start symbol so dlopen of aiehlc_ps.so fails; (2) runtime crash after "Loading kernel..." inside XAie_LoadElfMem → BlockWrite32 → MathEngine::try_sideband_fast_write → invalidate_operation_cache, caused by a num_tiles="0" Work_gen5/reports/aiehlc.xpe that makes the ISS disable all tiles ("Iss used row and col 0 0"). Use when a --platform sim run crashes at elaboration or during kernel load. --- # AIE Simulator PS.so Load-Crash Debug The AIE simulator (`aie2pssimmsm`) segfaults while **loading the PS shared library** (`script/sim/build/aiehlc_ps.so`), before any host/runtime code runs. Console shows only: ``` AIE_WORK_DIR = .../script/sim/Work_gen5 .../rdiArgs.sh: line 60: Segmentation fault (core dumped) "$RDI_PROG" "$@" segfault in .../unwrapped/lnx64.o/aie2pssimmsm ... ``` There is **no** `[aie_runtime] __Runtime_init--`, no `AIEHLC PS IP loaded`, no `AIEHLC PS IP started`. ## Key discriminator (run this first) Grep the sim log for the PS-thread markers: ```bash grep -aE 'AIEHLC PS IP started|AIEHLC PS IP loaded|__Runtime_init' ``` - **None present** → crash is at PS.so load / SystemC elaboration, *before* the PS thread runs. Continue below. - **Present, then crash later** → runtime crash. If the backtrace goes through `XAie_LoadElfMem` → `XAie_BlockWrite32` → `MathEngine::try_sideband_fast_write` → `invalidate_operation_cache`, see **Root cause #3** below. Otherwise use the DMA/data-mismatch tools. ## Root cause #3: crash in `XAie_LoadElfMem` — ISS disabled the target tile (`num_tiles="0"` xpe) **Symptom:** the sim reaches `AIEHLC PS IP started` and prints `Loading kernel 1...`, then segfaults inside the vendor model: ``` XAie_LoadElfMem → XAie_BlockWrite32 → PSIP_aiehlc::write32 → MathEngine::try_sideband_fast_write → me_sc::invalidate_operation_cache (SIGSEGV) ``` **Root cause:** the AIE2PS ISS reads `Work_gen5/reports/aiehlc.xpe` and uses its `num_tiles` to size the "used tile" grid. A stub xpe with `num_tiles="0"` makes the ISS decide **no** core tiles are used — the log shows `ISS disables unused tiles` then a single `Iss used row and col 0 0`. The runtime then does `XAie_LoadElfMem` (a PM `BlockWrite32`) into a **disabled** tile whose core model / operation cache was never constructed → segfault. (Independent of Vitis version — reproduces identically on 2026.1 and 2026.2.) ### Confirm ```bash grep -nE 'Found xpe|Iss used row and col|disables unused' cat script/sim/Work_gen5/reports/aiehlc.xpe # look for num_tiles="0" ``` - **Broken:** exactly one `Iss used row and col 0 0` (grid collapsed to origin). - **Healthy:** many lines — `Iss used row and col 0 0`, `0 1`, … `0 35`, `1 0`, … (the ISS enumerates the whole array). ### Fix Do **not** emit a `num_tiles="0"` xpe from `script/sim/gen_work_package.sh`. Removing the xpe block entirely is correct: without the file the ISS enables the full array and the ELF load succeeds. The `Summary File: Warning - Can't find aie compiler summary file!` warning is unrelated and appears with or without the xpe. Verify with `tutorial/example.cpp --platform sim`: expect `CPU=100..119 == AIE=100..119`, `Kernel test passed!`, `Sim result: 0`. ## Root cause #1 (most common): stale `kernel_elf_init.cc` `script/sim/build/kernel_elf_init.cc` is generated by [script/sim/Makefile](../../../script/sim/Makefile) and declares the embedded-kernel symbol `_binary_kernel__start/_end`, registering it via a `__attribute__((constructor))`. The kernel binary is embedded under the **actual** kernel name (e.g. `matmul`, `copyk`), but if `kernel_elf_init.cc` was generated for a **different** kernel (e.g. `loop_kernel` from a prior `tutorial/example.cpp` sim), it references a symbol that isn't in the `.so`. An unresolved `_binary_kernel__start` makes the simulator's `dlopen(aiehlc_ps.so)` fail; the simulator then segfaults on the NULL handle. **Why it happens:** the old Makefile target used an order-only prerequisite (`| $(BUILD_DIR)`) so `kernel_elf_init.cc` was generated once and **never regenerated** when the kernel changed. The single-kernel path (`aiehlc.sh`) removed it before each build, but the tiling path (`hostcompile.sh` → `runsim.sh`) did not. Clean builds (`clean_build.sh` removes `kernel_elf_init.{cc,o}`) dodged it; incremental builds after switching programs hit it. ### Confirm ```bash # 1. What symbol does kernel_elf_init.cc reference? grep -aoE '_binary_kernel_[A-Za-z0-9_]+_start' script/sim/build/kernel_elf_init.cc | head -1 # 2. What kernel is actually embedded in the PS.so? (D = defined) nm script/sim/build/aiehlc_ps.so | grep -E '_binary_kernel_.*_(start|end)' # 3. Undefined _binary_kernel_* symbols in the PS.so (U = unresolved => crash) nm -D -u script/sim/build/aiehlc_ps.so | grep -E '_binary_kernel_' ``` If (1) names a kernel that (3) shows as **U** (undefined) while (2) shows a **different** kernel as **D** (defined), this is the bug. Cross-check by diffing the crashing PS.so against a working single-kernel PS.so: ```bash nm -D -u /aiehlc_ps.so | awk '{print $NF}' | sort -u > /tmp/a nm -D -u /aiehlc_ps.so | awk '{print $NF}' | sort -u > /tmp/b comm -23 /tmp/a /tmp/b # symbols the crashing one needs but the working one doesn't ``` A lone `_binary_kernel__start/_end` in the output confirms it. ### Fix The Makefile must regenerate `kernel_elf_init.cc` whenever `KERNEL_FUNCNAME` changes. The committed fix uses a `FORCE` prerequisite that rewrites into a temp and swaps only when the content changed (regenerates on kernel change, no recompile churn otherwise): ```make .PHONY: FORCE FORCE: $(BUILD_DIR)/kernel_elf_init.cc: FORCE | $(BUILD_DIR) @printf '... _binary_kernel_$(if $(KERNEL_FUNCNAME),$(KERNEL_FUNCNAME)_,)start ...' > $@.tmp @if ! cmp -s $@.tmp $@ 2>/dev/null; then mv $@.tmp $@; echo "[gen] kernel_elf_init.cc"; else rm -f $@.tmp; fi ``` Immediate unblock without editing the Makefile: `rm -f script/sim/build/kernel_elf_init.{cc,o} script/sim/build/aiehlc_ps.so` before rebuilding, or run `clean_build.sh`. **Verify** the fix: after rebuild, `grep _binary_kernel script/sim/build/kernel_elf_init.cc` must match the actual kernel, the sim must print `AIEHLC PS IP started`, and reach `Simulation Finished, Sim result: 0`. ## Other load-time causes to check (if #1 is not it) Compare the crashing tiling PS.so to a working single-kernel PS.so (build `tutorial/example.cpp` with `--platform sim` as the known-good baseline): - **Undefined symbols:** `nm -D -u aiehlc_ps.so` — any unresolved symbol the simulator can't provide fails `dlopen`. - **Global constructors:** `nm aiehlc_ps.so | grep _GLOBAL__sub_I_` and `objdump -h aiehlc_ps.so | grep init_array` — a constructor that runs at load and crashes. - **NEEDED libs:** `objdump -p aiehlc_ps.so | grep NEEDED` — a missing/incompatible dependency. - **Missing sim stub:** `--sim-tiles` (i.e. `--stub-tiles`) needs `script/sim/build/stub_kernel_build/stub_kernel`, built from `example/aiesim_test/stub_kernel.cc`. That directory is **untracked in git** — absent on clean checkouts. runsim.sh must build the stub under `--stub-tiles`, not only `--stub-all`. ## Pitfalls / do-not-waste-time-on - **gdb is unusable on this simulator.** `gdb` (v12) mis-handles the vendor `libxv_isl_iostreams.so` ELF layout and crashes the *working* `tutorial/example.cpp` identically at startup — any backtrace you get is a gdb artifact, not the real crash. Diagnose via `nm`/`objdump` on the PS.so and the console PS-marker discriminator instead. - The crash is **independent of** kernel body, dead kernels, ping-pong (`pp_depth`), tile count, NoC/shim column, partition coords, and aie-rt version — don't chase these. Confirmed by bisecting to a 1-in-1-out copy tiling app (`example/tileprogram/ccode/mini_tile.cc`), which crashes identically until `kernel_elf_init.cc` is correct. - This is **not** rebase/branch-specific — it is a build-flow (stale generated file) bug. ## Minimal reproduction / smoke test `example/tileprogram/ccode/mini_tile.cc` is a ~50-line 1-input/1-output tiling copy. Run it after a single-kernel sim (without cleaning) to reproduce, and after the fix to confirm: ```bash source script/aiehlc.sh --platform sim --aie-version 5 --sim-tiles 0:3 \ --runtime-source-file ./example/tileprogram/ccode/mini_tile.cc bash script/runsim.sh aout/ ```