Decode And Triage

HRR Decode & Triage

Platform (Linux): AMD GPU host. Full skill workflow via triage_archive.sh: native GPU replay (auto-builds hrr-playback when missing), optional Docker replay, manifest preflight, and finding output.

Platform (Windows): Full native GPU replay via triage_archive.ps1 + ensure_playback.ps1 (PowerShell). Docker replay is not supported on Windows (ROCm containers are Linux-only) — see Windows support below.

Run from: any working directory. On Linux, invoke by absolute path:

<rocm-systems>/projects/hrr/skills/decode-and-triage/scripts/triage_archive.sh \
  --archive <path-to>/capture.hrr/pid-<pid>

On Windows, invoke by absolute path:

<rocm-systems>\projects\hrr\skills\decode-and-triage\scripts\triage_archive.ps1 `
  --archive <path-to>\capture.hrr\pid-<pid>

Scripts locate the colocated projects/hrr project from their install path; you do not need to cd into rocm-systems first.

Print the finding markdown that triage_archive.sh / triage_archive.ps1 writes to stdout.

Windows support

On Windows, prefer triage_archive.ps1 for full GPU replay. Fall back to --no-replay (metadata-only) when no GPU is available or hrr-playback.exe is absent and cannot be built.

W1 — Locate or build hrr-playback.exe

# From the skill scripts directory:
.\ensure_playback.ps1          # find existing binary (checks HRR_PLAYBACK, HIP_PATH\bin, ROCM_PATH\bin, PATH)
.\ensure_playback.ps1 --build  # also build hrr-playback from projects/hrr if not found

Set $env:HRR_PLAYBACK to the path printed, or let triage_archive.ps1 call it automatically.

W2 — Run triage (native GPU replay)

# Metadata only (no GPU required):
.\triage_archive.ps1 --archive C:\path\to\capture.hrr\pid-138 --no-replay

# Full native replay:
.\triage_archive.ps1 --archive C:\path\to\capture.hrr\pid-138

# Override GPU ordinal:
$env:GPU = "1"
.\triage_archive.ps1 --archive C:\path\to\capture.hrr\pid-138

Environment variables on Windows:

W3 — Docker replay on Windows

Docker GPU replay (--device=/dev/kfd) is Linux-only. On Windows:

• Use WSL2 (Ubuntu 22.04+) and run replay_docker.sh inside WSL2:

  # In WSL2 shell:
  export HRR_DOCKER_IMAGE="rocm/vllm:rocm7.13.0_gfx950-dcgpu_ubuntu24.04_py3.13_pytorch_2.10.0_vllm_0.19.1"
  bash scripts/replay_docker.sh \
    --archive /mnt/c/path/to/capture.hrr/pid-138 \
    --log /tmp/hrr.log
  

• Or use Cursor Remote SSH → Linux ROCm server and run the Linux skill there.

W4 — Print finding summary

The .finding.md written by triage_archive.ps1 is identical in schema to the Linux version. Print or paste it directly in the chat reply using the same template in Step 5 below.

Copying scripts to a remote machine (no Cursor installed)

When the HIP app runs on a Remote Desktop machine without Cursor, copy the four standalone script files and run them directly:

projects\hrr\skills\decode-and-triage\scripts\
  triage_archive.ps1
  ensure_playback.ps1
  analyze_replay_finding.py
  check_replay_compat.py

Requirements on the remote: PowerShell 5+, Python 3.8+, and hrr-playback.exe (from HIP SDK bin\ or built via ensure_playback.ps1 --build).

Bring the resulting .finding.md back and paste it into the chat for AI analysis.

Example prompt

Use the decode-and-triage skill. Triage with docker replay:
<path-to>/capture.hrr/pid-<pid>

Docker image used for capture:
<registry>/<image>:<tag>

Workflow

Replay mode (auto default): picks docker when HRR_DOCKER_IMAGE is set, else native. Use --no-replay for metadata-only (no GPU).

1. Resolve capture.hrr/pid-* (largest events.bin if user gives root only).
2. Preflight when manifest.json has metadata: block if capture used more GPUs than replay exposes or requested GPU is missing. HIP/comgr mismatch prompts for confirmation (exit 2 in agents — ask user, then HRR_CONTINUE=1). Docker preflight uses the image HRR stack by default; HRR_DOCKER_MOUNT_CLR=1 overlays host CLR_BUILD/HRR_PLAYBACK for WIP dev work. Legacy archives skip preflight. Overrides: HRR_SKIP_COMPAT=1, HRR_STRICT_VERSION=1, HRR_STRICT_ARCH=1.
3. Native (Linux): triage_archive.sh --archive <pid-dir> (builds/finds host hrr-playback).
4. Native (Windows): triage_archive.ps1 --archive <pid-dir> (finds/builds hrr-playback.exe).
5. Docker (Linux only): export HRR_DOCKER_IMAGE='<capture image>' then triage_archive.sh --archive <pid-dir> --replay docker. Image hrr-playback by default; dev overlay via HRR_DOCKER_MOUNT_CLR=1. Optional GPU=1 when GPU 0 busy.
6. Print the finding summary from triage output. Add a one-line capture explainer if helpful.

Finding outcomes

When outcome is UNKNOWN or fault class is unknown, say so explicitly in the summary — do not invent a fault type.

A fault on the GPU and a crash of the replay process are different findings. The runtime's Memory access fault by GPU node-N ... on address 0x... line is the GPU one: outcome MAF, fault class illegal_memory_access, or read_only_page_fault when the reason names a read-only page. The replay process dying on a signal without printing any of that is the host one: outcome ABORT, fault class replay_crashed, and it implicates no kernel, because nothing about the workload was established. Report that as a failed run rather than as the user's defect.

An HSA queue abort (HSA_STATUS_ERROR_EXCEPTION, ABORTED, MEMORY_FAULT) is not a hang: MEMORY_FAULT is a fault and the other two are hardware exceptions, which an out-of-bounds access raises. The abort line carries the kernel but no faulting address, so read the kernel from there. A genuine hang shows up as no progress against the clock, which is what HRR_REPLAY_TIMEOUT reports; do not report one on the strength of an HSA status alone.

A faulting ATen kernel needs the original failure signature

PyTorch and vLLM reach the GPU through <<<>>> (hipLaunchByPtr), and those kernels pass device pointers inside by-value structs: vectorized_elementwise_kernel takes a std::array<char*,N>, reduce_kernel a config struct with pointers at arbitrary offsets. Capture records those offsets and replay translates them, with a defensive rescan for any the capture-time heuristic missed, so on a current build these kernels replay faithfully and a fault on one is a real finding.

Two things still make such a fault ambiguous: the detector is a value-based heuristic, and an archive recorded before that support landed carries no offsets at all, in which case replay launches with a capture-time address and faults on a workload that is fine. Both look identical to a workload defect.

So the fault class stays what the evidence says (illegal_memory_access), the finding carries a note, and you ask the user for their original failure signature before calling it their bug. If their run never faulted here, suspect the recording. Faults on hipModuleLaunchKernel-launched kernels (hipBLASLt GEMMs, custom HIP kernels) do not carry this ambiguity.

Kernel attribution

Replay runs with --sync-after-launch, which triage_archive.sh adds for you. Without it the GPU is serialized only once at the end, so a fault is reported but not attributed and the finding has no failing event and no kernel. Pass --no-sync only when throughput matters more than attribution.

A clean replay implicates no kernel, so the finding reports none even though the archive lists the kernels it ran. Never present a kernel next to a PASS.

A memory fault can tear the process down before the failing dispatch is attributed, leaving a log with no kernel at all. When the archive holds exactly one kernel, the finding names it and says the name was inferred; confirm it with --sync-after-launch, which stops replay on the faulting dispatch. With more than one kernel it stays unknown rather than guessing. Names that --info truncated to its column width are never used: a cut-off symbol cannot be looked up or handed to a kernel developer.

Finding summary template (Step 5 / W4)

Print a structured summary directly in the chat reply:

## HRR Triage — <archive name>

### Archive health
| Field           | Value |
|-----------------|-------|
| Complete        | yes / NO |
| Recovered tail  | <n> events (0 = clean) |
| Processes       | <pids> |
| Events          | <n> |
| Blobs           | <n> |

### Replay result
| Field           | Value |
|-----------------|-------|
| Exit code       | 0 / 1 / 2 |
| D2H pass        | <n> exact + <n> within-tol |
| D2H fail        | <n> |
| Kernel GPU time | <ms> ms |
| Graph GPU time  | <ms> ms |
| Total GPU time  | <ms> ms |

### Findings
- **F1** <finding> — <one-line explanation>
- **F2** ...

### Recommended next steps
1. ...

Classify each finding using these labels:

• CRASH-TRUNCATED — archive missing clean trailer; replay may be partial
• D2H-FAIL — output mismatch; note which blobs and max error if logged
• MISSING-HANDLE — null-translated pointer; likely hipHostAlloc/texture/IPC gap
• NOOP-API — explicit graph, texture object, or symbol API replayed as no-op
• BUILD-NEEDED — hrr-playback was absent and had to be built
• MULTI-PROCESS — root archive spans multiple pid sub-archives; triage each separately
• CLEAN — no issues found

Hard constraints

• Windows hosts: use triage_archive.ps1 for full GPU triage. Do not run triage_archive.sh for full GPU replay unless the user explicitly has bash + a working Linux-style GPU path. Prefer triage_archive.ps1 --no-replay (or analyze_replay_finding.py --archive <pid-dir>) when no GPU is available.
• No source edits. One ensure_playback.sh --build / ensure_playback.ps1 --build attempt max; on failure stop (or use --no-replay).
• Do not ask about Docker until native replay fails unless user named the capture image.
• Do not ask user to run cmake/ninja — scripts handle it.
• Do not read stale *.finding.md / hrr-replay-*.log unless user points at them.
• Do not modify the archive directory or its contents.
• Do not run hrr-playback --repair unless the user explicitly asks.

If build or replay fails

Quick reference — hrr-playback flags

Known limitations to flag automatically

If these patterns appear in replay output, add the corresponding finding: