decode-and-triage

# HRR Decode & Triage **Platform (Linux):** AMD GPU host. Full skill workflow via `triage_archive.sh`: native GPU replay (auto-builds `hrr-playback` when missing), optional Docker replay, manifest preflight, and finding output. **Platform (Windows):** Full native GPU replay via `triage_archive.ps1` + `ensure_playback.ps1` (PowerShell). Docker replay is **not** supported on Windows (ROCm containers are Linux-only) — see [Windows support](#windows-support) below. **Run from:** any working directory. On Linux, invoke by absolute path: ```bash <rocm-systems>/projects/hrr/skills/decode-and-triage/scripts/triage_archive.sh \ --archive <path-to>/capture.hrr/pid-<pid> ``` On Windows, invoke by absolute path: ```powershell <rocm-systems>\projects\hrr\skills\decode-and-triage\scripts\triage_archive.ps1 ` --archive <path-to>\capture.hrr\pid-<pid> ``` Scripts locate the colocated `projects/hrr` project from their install path; you do not need to `cd` into `rocm-systems` first. Print the **finding markdown** that `triage_archive.sh` / `triage_archive.ps1` writes to stdout. ### Windows support | Capability | Windows today | |------------|---------------| | Full skill via `triage_archive.ps1` | **Yes** — PowerShell port with native GPU replay, manifest preflight, and `.finding.md` output | | Locate / build `hrr-playback.exe` | **Yes** — `ensure_playback.ps1` searches `HRR_PLAYBACK`, `HIP_PATH\bin`, `ROCM_PATH\bin`, `PATH`; `--build` builds `hrr-playback` from `projects/hrr` | | Metadata-only triage (`--no-replay`) | **Yes** — `triage_archive.ps1 --no-replay` or `python analyze_replay_finding.py --archive <pid-dir>` | | Native GPU replay | **Yes** — `triage_archive.ps1` launches `hrr-playback.exe` via `Start-Process` with stdout/stderr capture | | Auto-build `hrr-playback.exe` | **Yes** — `ensure_playback.ps1 --build` builds `hrr-playback` from `projects/hrr` into a dedicated HRR build dir | | Docker replay (`--replay docker`) | **No** — requires a Linux host with Docker and GPU passthrough; use WSL2 or a Linux SSH host | | Container captures (e.g. vLLM image) | Replay on **Linux** (native or Docker); on Windows use `triage_archive.ps1` native replay if the Windows HIP stack matches | On Windows, prefer `triage_archive.ps1` for full GPU replay. Fall back to `--no-replay` (metadata-only) when no GPU is available or `hrr-playback.exe` is absent and cannot be built. #### W1 — Locate or build `hrr-playback.exe` ```powershell # From the skill scripts directory: .\ensure_playback.ps1 # find existing binary (checks HRR_PLAYBACK, HIP_PATH\bin, ROCM_PATH\bin, PATH) .\ensure_playback.ps1 --build # also build hrr-playback from projects/hrr if not found ``` Set `$env:HRR_PLAYBACK` to the path printed, or let `triage_archive.ps1` call it automatically. #### W2 — Run triage (native GPU replay) ```powershell # Metadata only (no GPU required): .\triage_archive.ps1 --archive C:\path\to\capture.hrr\pid-138 --no-replay # Full native replay: .\triage_archive.ps1 --archive C:\path\to\capture.hrr\pid-138 # Override GPU ordinal: $env:GPU = "1" .\triage_archive.ps1 --archive C:\path\to\capture.hrr\pid-138 ``` Environment variables on Windows: | Variable | Purpose | |---|---| | `HRR_PLAYBACK` | Explicit `hrr-playback.exe` path | | `HIP_PATH` or `ROCM_PATH` | HIP SDK root (default: `C:\Program Files\AMD\ROCm\6.2`) | | `HRR_BUILD` | Dedicated HRR build dir for `--build` (default: `projects/hrr\build-playback`) | | `GPU` | GPU ordinal for replay (default: `0`) | | `HRR_TRIAGE_WORKDIR` | Output dir for findings + logs | | `HRR_CONTINUE=1` | Proceed past HIP/comgr version mismatch | | `HRR_SKIP_COMPAT=1` | Skip manifest preflight entirely | #### W3 — Docker replay on Windows Docker GPU replay (`--device=/dev/kfd`) is **Linux-only**. On Windows: - Use WSL2 (Ubuntu 22.04+) and run `replay_docker.sh` inside WSL2: ```bash # In WSL2 shell: export HRR_DOCKER_IMAGE="rocm/vllm:rocm7.13.0_gfx950-dcgpu_ubuntu24.04_py3.13_pytorch_2.10.0_vllm_0.19.1" bash scripts/replay_docker.sh \ --archive /mnt/c/path/to/capture.hrr/pid-138 \ --log /tmp/hrr.log ``` - Or use Cursor Remote SSH → Linux ROCm server and run the Linux skill there. #### W4 — Print finding summary The `.finding.md` written by `triage_archive.ps1` is identical in schema to the Linux version. Print or paste it directly in the chat reply using the same template in Step 5 below. #### Copying scripts to a remote machine (no Cursor installed) When the HIP app runs on a Remote Desktop machine without Cursor, copy the four standalone script files and run them directly: ``` projects\hrr\skills\decode-and-triage\scripts\ triage_archive.ps1 ensure_playback.ps1 analyze_replay_finding.py check_replay_compat.py ``` Requirements on the remote: PowerShell 5+, Python 3.8+, and `hrr-playback.exe` (from HIP SDK `bin\` or built via `ensure_playback.ps1 --build`). Bring the resulting `.finding.md` back and paste it into the chat for AI analysis. --- ## Example prompt ``` Use the decode-and-triage skill. Triage with docker replay: <path-to>/capture.hrr/pid-<pid> Docker image used for capture: <registry>/<image>:<tag> ``` ## Workflow **Replay mode (`auto` default):** picks docker when `HRR_DOCKER_IMAGE` is set, else native. Use `--no-replay` for metadata-only (no GPU). 1. Resolve `capture.hrr/pid-*` (largest `events.bin` if user gives root only). 2. **Preflight** when `manifest.json` has `metadata`: block if capture used more GPUs than replay exposes or requested `GPU` is missing. HIP/comgr mismatch prompts for confirmation (exit 2 in agents — ask user, then `HRR_CONTINUE=1`). Docker preflight uses the **image HRR stack** by default; `HRR_DOCKER_MOUNT_CLR=1` overlays host `CLR_BUILD`/`HRR_PLAYBACK` for WIP dev work. Legacy archives skip preflight. Overrides: `HRR_SKIP_COMPAT=1`, `HRR_STRICT_VERSION=1`, `HRR_STRICT_ARCH=1`. 3. **Native (Linux):** `triage_archive.sh --archive <pid-dir>` (builds/finds host `hrr-playback`). 4. **Native (Windows):** `triage_archive.ps1 --archive <pid-dir>` (finds/builds `hrr-playback.exe`). 5. **Docker (Linux only):** `export HRR_DOCKER_IMAGE='<capture image>'` then `triage_archive.sh --archive <pid-dir> --replay docker`. Image `hrr-playback` by default; dev overlay via `HRR_DOCKER_MOUNT_CLR=1`. Optional `GPU=1` when GPU 0 busy. 6. Print the finding summary from triage output. Add a one-line capture explainer if helpful. ### Finding outcomes | Outcome | Meaning | |---------|---------| | `PASS` | Replay completed; D2H checks passed | | `MAF` | GPU memory access fault during replay | | `FAIL` | Replay finished but validation failed (e.g. D2H mismatch) | | `ABORT` | Replay stopped early (fatal API, version mismatch, queue abort, user abort, replay process killed by a signal) | | `HANG` | Replay made no progress and the runner had to stop it (see `HRR_REPLAY_TIMEOUT`) | | `UNKNOWN` | Insufficient signal to classify (e.g. metadata-only triage, missing log) | When outcome is `UNKNOWN` or fault class is `unknown`, say so explicitly in the summary — do not invent a fault type. A fault on the GPU and a crash of the replay process are different findings. The runtime's `Memory access fault by GPU node-N ... on address 0x...` line is the GPU one: outcome `MAF`, fault class `illegal_memory_access`, or `read_only_page_fault` when the reason names a read-only page. The replay process dying on a signal without printing any of that is the host one: outcome `ABORT`, fault class `replay_crashed`, and it implicates no kernel, because nothing about the workload was established. Report that as a failed run rather than as the user's defect. An HSA queue abort (`HSA_STATUS_ERROR_EXCEPTION`, `ABORTED`, `MEMORY_FAULT`) is not a hang: `MEMORY_FAULT` is a fault and the other two are hardware exceptions, which an out-of-bounds access raises. The abort line carries the kernel but no faulting address, so read the kernel from there. A genuine hang shows up as no progress against the clock, which is what `HRR_REPLAY_TIMEOUT` reports; do not report one on the strength of an HSA status alone. ### A faulting ATen kernel needs the original failure signature PyTorch and vLLM reach the GPU through `<<<>>>` (`hipLaunchByPtr`), and those kernels pass device pointers inside by-value structs: `vectorized_elementwise_kernel` takes a `std::array<char*,N>`, `reduce_kernel` a config struct with pointers at arbitrary offsets. Capture records those offsets and replay translates them, with a defensive rescan for any the capture-time heuristic missed, so on a current build these kernels replay faithfully and a fault on one is a real finding. Two things still make such a fault ambiguous: the detector is a value-based heuristic, and an archive recorded before that support landed carries no offsets at all, in which case replay launches with a capture-time address and faults on a workload that is fine. Both look identical to a workload defect. So the fault class stays what the evidence says (`illegal_memory_access`), the finding carries a note, and you **ask the user for their original failure signature** before calling it their bug. If their run never faulted here, suspect the recording. Faults on `hipModuleLaunchKernel`-launched kernels (hipBLASLt GEMMs, custom HIP kernels) do not carry this ambiguity. ### Kernel attribution Replay runs with `--sync-after-launch`, which `triage_archive.sh` adds for you. Without it the GPU is serialized only once at the end, so a fault is reported but not attributed and the finding has no failing event and no kernel. Pass `--no-sync` only when throughput matters more than attribution. A clean replay implicates no kernel, so the finding reports none even though the archive lists the kernels it ran. Never present a kernel next to a `PASS`. A memory fault can tear the process down before the failing dispatch is attributed, leaving a log with no kernel at all. When the archive holds exactly one kernel, the finding names it and says the name was inferred; confirm it with `--sync-after-launch`, which stops replay on the faulting dispatch. With more than one kernel it stays unknown rather than guessing. Names that `--info` truncated to its column width are never used: a cut-off symbol cannot be looked up or handed to a kernel developer. ### Finding summary template (Step 5 / W4) Print a structured summary directly in the chat reply: ``` ## HRR Triage — <archive name> ### Archive health | Field | Value | |-----------------|-------| | Complete | yes / NO | | Recovered tail | <n> events (0 = clean) | | Processes | <pids> | | Events | <n> | | Blobs | <n> | ### Replay result | Field | Value | |-----------------|-------| | Exit code | 0 / 1 / 2 | | D2H pass | <n> exact + <n> within-tol | | D2H fail | <n> | | Kernel GPU time | <ms> ms | | Graph GPU time | <ms> ms | | Total GPU time | <ms> ms | ### Findings - **F1** <finding> — <one-line explanation> - **F2** ... ### Recommended next steps 1. ... ``` Classify each finding using these labels: - **CRASH-TRUNCATED** — archive missing clean trailer; replay may be partial - **D2H-FAIL** — output mismatch; note which blobs and max error if logged - **MISSING-HANDLE** — null-translated pointer; likely `hipHostAlloc`/texture/IPC gap - **NOOP-API** — explicit graph, texture object, or symbol API replayed as no-op - **BUILD-NEEDED** — `hrr-playback` was absent and had to be built - **MULTI-PROCESS** — root archive spans multiple pid sub-archives; triage each separately - **CLEAN** — no issues found ## Hard constraints - **Windows hosts:** use `triage_archive.ps1` for full GPU triage. Do not run `triage_archive.sh` for full GPU replay unless the user explicitly has bash + a working Linux-style GPU path. Prefer `triage_archive.ps1 --no-replay` (or `analyze_replay_finding.py --archive <pid-dir>`) when no GPU is available. - **No source edits.** One `ensure_playback.sh --build` / `ensure_playback.ps1 --build` attempt max; on failure stop (or use `--no-replay`). - Do not ask about Docker until native replay fails **unless** user named the capture image. - Do not ask user to run cmake/ninja — scripts handle it. - Do not read stale `*.finding.md` / `hrr-replay-*.log` unless user points at them. - Do **not** modify the archive directory or its contents. - Do **not** run `hrr-playback --repair` unless the user explicitly asks. ## If build or replay fails | Situation | Action | |-----------|--------| | `--build` fails | Install `hrr-playback` under `$ROCM_PATH/bin` or set `HRR_PLAYBACK`; ensure a capture-enabled ROCm/HIP prefix (see README build section). | | Native replay fails (Linux) | Docker replay with capture image (Linux host). | | Native replay fails (Windows) | Check `HRR_PLAYBACK`, `HIP_PATH\bin` on `PATH`; use `--no-replay` as fallback. | | Windows host, no GPU | Metadata-only via `triage_archive.ps1 --no-replay` or `analyze_replay_finding.py --archive`. | | Docker `hrr-playback` not in image | `HRR_DOCKER_MOUNT_CLR=1` with `CLR_BUILD` / `HRR_PLAYBACK`. |