Aiesimloaddebug

name: aiesimloaddebug description: Debug AIE simulator (aie2pssimmsm) segfaults. Two families — (1) load-time crash at PS.so load, before "AIEHLC PS IP started", usually a stale script/sim/build/kernel_elf_init.cc referencing the wrong binary_kernel_start symbol so dlopen of aiehlc_ps.so fails; (2) runtime crash after "Loading kernel..." inside XAie_LoadElfMem → BlockWrite32 → MathEngine::try_sideband_fast_write → invalidate_operation_cache, caused by a num_tiles="0" Work_gen5/reports/aiehlc.xpe that makes the ISS disable all tiles ("Iss used row and col 0 0"). Use when a --platform sim run crashes at elaboration or during kernel load.

AIE Simulator PS.so Load-Crash Debug

The AIE simulator (aie2pssimmsm) segfaults while loading the PS shared library (script/sim/build/aiehlc_ps.so), before any host/runtime code runs. Console shows only:

AIE_WORK_DIR = .../script/sim/Work_gen5
.../rdiArgs.sh: line 60: <pid> Segmentation fault (core dumped) "$RDI_PROG" "$@"
segfault in .../unwrapped/lnx64.o/aie2pssimmsm ...

There is no [aie_runtime] __Runtime_init--, no AIEHLC PS IP loaded, no AIEHLC PS IP started.

Key discriminator (run this first)

Grep the sim log for the PS-thread markers:

grep -aE 'AIEHLC PS IP started|AIEHLC PS IP loaded|__Runtime_init' <simlog>

• None present → crash is at PS.so load / SystemC elaboration, before the PS thread runs. Continue below.
• Present, then crash later → runtime crash. If the backtrace goes through XAie_LoadElfMem → XAie_BlockWrite32 → MathEngine::try_sideband_fast_write → invalidate_operation_cache, see Root cause #3 below. Otherwise use the DMA/data-mismatch tools.

Root cause #3: crash in XAie_LoadElfMem — ISS disabled the target tile (num_tiles="0" xpe)

Symptom: the sim reaches AIEHLC PS IP started and prints Loading kernel 1..., then segfaults inside the vendor model:

XAie_LoadElfMem → XAie_BlockWrite32 → PSIP_aiehlc::write32
  → MathEngine::try_sideband_fast_write → me_sc::invalidate_operation_cache  (SIGSEGV)

Root cause: the AIE2PS ISS reads Work_gen5/reports/aiehlc.xpe and uses its num_tiles to size the "used tile" grid. A stub xpe with num_tiles="0" makes the ISS decide no core tiles are used — the log shows ISS disables unused tiles then a single Iss used row and col 0 0. The runtime then does XAie_LoadElfMem (a PM BlockWrite32) into a disabled tile whose core model / operation cache was never constructed → segfault. (Independent of Vitis version — reproduces identically on 2026.1 and 2026.2.)

Confirm

grep -nE 'Found xpe|Iss used row and col|disables unused' <simlog>
cat script/sim/Work_gen5/reports/aiehlc.xpe   # look for num_tiles="0"

• Broken: exactly one Iss used row and col 0 0 (grid collapsed to origin).
• Healthy: many lines — Iss used row and col 0 0, 0 1, … 0 35, 1 0, … (the ISS enumerates the whole array).

Fix

Do not emit a num_tiles="0" xpe from script/sim/gen_work_package.sh. Removing the xpe block entirely is correct: without the file the ISS enables the full array and the ELF load succeeds. The Summary File: Warning - Can't find aie compiler summary file! warning is unrelated and appears with or without the xpe. Verify with tutorial/example.cpp --platform sim: expect CPU=100..119 == AIE=100..119, Kernel test passed!, Sim result: 0.

Root cause #1 (most common): stale kernel_elf_init.cc

script/sim/build/kernel_elf_init.cc is generated by script/sim/Makefile and declares the embedded-kernel symbol _binary_kernel_<KERNEL_FUNCNAME>_start/_end, registering it via a __attribute__((constructor)). The kernel binary is embedded under the actual kernel name (e.g. matmul, copyk), but if kernel_elf_init.cc was generated for a different kernel (e.g. loop_kernel from a prior tutorial/example.cpp sim), it references a symbol that isn't in the .so.

An unresolved _binary_kernel_<wrong>_start makes the simulator's dlopen(aiehlc_ps.so) fail; the simulator then segfaults on the NULL handle.

Why it happens: the old Makefile target used an order-only prerequisite (| $(BUILD_DIR)) so kernel_elf_init.cc was generated once and never regenerated when the kernel changed. The single-kernel path (aiehlc.sh) removed it before each build, but the tiling path (hostcompile.sh → runsim.sh) did not. Clean builds (clean_build.sh removes kernel_elf_init.{cc,o}) dodged it; incremental builds after switching programs hit it.

Confirm

# 1. What symbol does kernel_elf_init.cc reference?
grep -aoE '_binary_kernel_[A-Za-z0-9_]+_start' script/sim/build/kernel_elf_init.cc | head -1

# 2. What kernel is actually embedded in the PS.so? (D = defined)
nm script/sim/build/aiehlc_ps.so | grep -E '_binary_kernel_.*_(start|end)'

# 3. Undefined _binary_kernel_* symbols in the PS.so (U = unresolved => crash)
nm -D -u script/sim/build/aiehlc_ps.so | grep -E '_binary_kernel_'

If (1) names a kernel that (3) shows as U (undefined) while (2) shows a different kernel as D (defined), this is the bug.

Cross-check by diffing the crashing PS.so against a working single-kernel PS.so:

nm -D -u <crashing>/aiehlc_ps.so | awk '{print $NF}' | sort -u > /tmp/a
nm -D -u <working>/aiehlc_ps.so  | awk '{print $NF}' | sort -u > /tmp/b
comm -23 /tmp/a /tmp/b   # symbols the crashing one needs but the working one doesn't

A lone _binary_kernel_<name>_start/_end in the output confirms it.

Fix

The Makefile must regenerate kernel_elf_init.cc whenever KERNEL_FUNCNAME changes. The committed fix uses a FORCE prerequisite that rewrites into a temp and swaps only when the content changed (regenerates on kernel change, no recompile churn otherwise):

.PHONY: FORCE
FORCE:
$(BUILD_DIR)/kernel_elf_init.cc: FORCE | $(BUILD_DIR)
	@printf '... _binary_kernel_$(if $(KERNEL_FUNCNAME),$(KERNEL_FUNCNAME)_,)start ...' > $@.tmp
	@if ! cmp -s $@.tmp $@ 2>/dev/null; then mv $@.tmp $@; echo "[gen] kernel_elf_init.cc"; else rm -f $@.tmp; fi

Immediate unblock without editing the Makefile: rm -f script/sim/build/kernel_elf_init.{cc,o} script/sim/build/aiehlc_ps.so before rebuilding, or run clean_build.sh.

Verify the fix: after rebuild, grep _binary_kernel script/sim/build/kernel_elf_init.cc must match the actual kernel, the sim must print AIEHLC PS IP started, and reach Simulation Finished, Sim result: 0.

Other load-time causes to check (if #1 is not it)

Compare the crashing tiling PS.so to a working single-kernel PS.so (build tutorial/example.cpp with --platform sim as the known-good baseline):

• Undefined symbols: nm -D -u aiehlc_ps.so — any unresolved symbol the simulator can't provide fails dlopen.
• Global constructors: nm aiehlc_ps.so | grep _GLOBAL__sub_I_ and objdump -h aiehlc_ps.so | grep init_array — a constructor that runs at load and crashes.
• NEEDED libs: objdump -p aiehlc_ps.so | grep NEEDED — a missing/incompatible dependency.
• Missing sim stub: --sim-tiles (i.e. --stub-tiles) needs script/sim/build/stub_kernel_build/stub_kernel, built from example/aiesim_test/stub_kernel.cc. That directory is untracked in git — absent on clean checkouts. runsim.sh must build the stub under --stub-tiles, not only --stub-all.

Pitfalls / do-not-waste-time-on

• gdb is unusable on this simulator. gdb (v12) mis-handles the vendor libxv_isl_iostreams.so ELF layout and crashes the working tutorial/example.cpp identically at startup — any backtrace you get is a gdb artifact, not the real crash. Diagnose via nm/objdump on the PS.so and the console PS-marker discriminator instead.
• The crash is independent of kernel body, dead kernels, ping-pong (pp_depth), tile count, NoC/shim column, partition coords, and aie-rt version — don't chase these. Confirmed by bisecting to a 1-in-1-out copy tiling app (example/tileprogram/ccode/mini_tile.cc), which crashes identically until kernel_elf_init.cc is correct.
• This is not rebase/branch-specific — it is a build-flow (stale generated file) bug.

Minimal reproduction / smoke test

example/tileprogram/ccode/mini_tile.cc is a ~50-line 1-input/1-output tiling copy. Run it after a single-kernel sim (without cleaning) to reproduce, and after the fix to confirm:

source script/aiehlc.sh --platform sim --aie-version 5 --sim-tiles 0:3 \
    --runtime-source-file ./example/tileprogram/ccode/mini_tile.cc
bash script/runsim.sh aout/