verify-pmc-metrics

# Mock-Based PMC Validation Deterministic validation of AMD SMI Performance Monitoring Counter (PMC) metrics using the mock system (`AMD_SMI_MOCK_CONFIG`). Tests sentinel filtering, expected value ranges, positive presence of configured metrics, and negative absence of unsupported metrics across all output formats. <IMPORTANT> - Execute all profiling and validation automatically — do NOT ask for user confirmation. - Run ALL 6 test matrix entries (2 architectures x 3 formats). - Sentinel detection is THE critical check — a single leaked sentinel is a FAIL. - Use rocprof-sys MCP tools for Perfetto traces, sqlite3 for RocPD databases. - Mock configs are in the worktree at `~/worktree/rocm-systems/amd-smi-mocks/`. </IMPORTANT> ## Sentinel Value Reference GPU firmware returns these values for unsupported/unavailable metrics. They must NEVER appear in output. | Type | Hex | Decimal | Float (Perfetto) | Where Used | |------|-----|---------|-------------------|------------| | `uint16_t` | `0xFFFF` | 65535 | 65535.0 | Temperature, activity %, power, voltage, clocks, PCIe width/speed, VCN/JPEG arrays | | `uint32_t` | `0xFFFFFFFF` | 4294967295 | 4294967295.0 | Various 32-bit fields | | `uint64_t` | `0xFFFFFFFFFFFFFFFF` | 18446744073709551615 | 1.8446744073709552e+19 | PCIe bandwidth, XGMI data, memory | Perfetto stores counter values as doubles, so `UINT64_MAX` appears as `1.8446744073709552e+19`. Both exact and float representations must be checked. ## Known Sentinel Filtering Pipeline | Metric | device.hpp | perfetto_policy.hpp | perfetto_processor.cpp | rocpd_processor.cpp | |--------|-----------|--------------------|-----------------------|--------------------| | VCN/JPEG (uint16) | Support check only | Filtered | Filtered | **NOT filtered** | | XGMI (uint64) | Zeroed via `populate_if_supported()` | N/A | Double-checked | Filtered | | PCIe (uint16/64) | Zeroed via `populate_if_supported()` | N/A | Relies on zero check | **NOT filtered** | | NIC RDMA | N/A | N/A | No filtering | No filtering | **Known leakage risks:** 1. RocPD VCN/JPEG array values (no explicit sentinel filter) 2. RocPD PCIe values (relies on device.hpp zeroing only) 3. NIC metrics in all outputs (no sentinel filtering anywhere) ## Mock Configurations Two architectures test complementary metric subsets. Located in `~/worktree/rocm-systems/amd-smi-mocks/`. ### MI300X (`mock_mi300x.yaml`) Data-center GPU with per-XCP metrics, HBM, and NIC. | Category | What's Configured | Expected Range | Notes | |----------|------------------|----------------|-------| | Temperature | hotspot only | [55, 75] | No temperature_edge (sentineled) | | HBM Temperature | 4 active / 4 total | [40, 60] | Array: active entries have values, rest sentineled | | Power | current_socket_power | [300, 500] | Watts | | GFX Busy | gfx_activity | [80, 100] | % | | UMC Busy | umc_activity | [0, 10] | % | | MM Busy | NOT configured | N/A | Should be absent (sentineled) | | VCN Busy | per-XCP, 4 active engines | [0, 10] | Per-XCP format, NOT device-level VCN Activity | | JPEG Busy | per-XCP, 32 active engines | [0, 10] | Per-XCP format, NOT device-level JPEG Activity | | PCIe Width | Fixed 16 | 16 | Lanes | | PCIe Speed | Fixed 32000 | 32000 | MT/s | | XGMI | NOT configured | N/A | Should be absent (sentineled) | | SDMA | 2 processes | varies | Cumulative byte counters | | Memory | vram_used | varies | Derived from vram_used / vram_total | | Voltages | NOT configured | N/A | Should be absent (sentineled) | | Fan | NOT configured | N/A | Should be absent (sentineled) | | NIC RDMA | 6 stats configured | [0, 1000000] | RX/TX bytes, packets, CNP | ### Navi 48 (`mock_navi48.yaml`) Consumer GPU with device-level metrics, no XCP, no NIC. | Category | What's Configured | Expected Range | Notes | |----------|------------------|----------------|-------| | Temperature | edge, hotspot, mem | [35, 75] | Has temperature_edge | | HBM Temperature | NOT configured | N/A | Should be absent (sentineled) | | Power | current_socket_power | [20, 30] | Watts | | GFX Busy | gfx_activity | [0, 10] | % | | UMC Busy | umc_activity | [0, 5] | % | | MM Busy | mm_activity | [0, 10] | % (present, unlike MI300X) | | VCN Activity | device-level, 1 active / 4 total | [0, 10] | Device-level, NOT per-XCP | | JPEG Activity | NOT configured | N/A | Should be absent (sentineled) | | PCIe Width | Fixed 16 | 16 | Lanes | | PCIe Speed | Fixed 2500 | 2500 | MT/s | | XGMI | NOT configured | N/A | Should be absent | | SDMA | 1 process | varies | Cumulative byte counters | | Memory | vram_used | varies | Derived | | Voltages | gfx, soc, mem configured | [700, 1200] | mV | | Fan | rpm and max configured | [800, 1200] | RPM | | NIC | No NIC devices | N/A | No NIC tracks should appear | ## Output Formats | # | Format | Environment Variables | |---|--------|---------------------| | 1 | Perfetto Standard | `ROCPROFSYS_TRACE=1` | | 2 | Perfetto Legacy | `ROCPROFSYS_TRACE=1 ROCPROFSYS_TRACE_LEGACY=true` | | 3 | RocPD | `ROCPROFSYS_USE_ROCPD=1` | ### Perfetto Track Name Differences | Format | GPU Pattern | NIC Pattern | |--------|------------|-------------| | Standard | `GPU [N] <Metric> (S)` | `NIC [N] <Metric> (S)` | | Legacy | `GPU <Metric> [N] (S)` | `NIC <device_name> <Metric> [N] (S)` | Legacy also uses abbreviated forms: `Pkts` vs `Packets`. ## Test Matrix | Run ID | Config | Format | Extra Env | |--------|--------|--------|-----------| | mi300-perfetto | mock_mi300x.yaml | Perfetto Standard | `ROCPROFSYS_SAMPLING_AINICS="all"` | | mi300-legacy | mock_mi300x.yaml | Perfetto Legacy | `ROCPROFSYS_SAMPLING_AINICS="all"` | | mi300-rocpd | mock_mi300x.yaml | RocPD | `ROCPROFSYS_SAMPLING_AINICS="all"` | | navi-perfetto | mock_navi48.yaml | Perfetto Standard | — | | navi-legacy | mock_navi48.yaml | Perfetto Legacy | — | | navi-rocpd | mock_navi48.yaml | RocPD | — | Common env for ALL runs: - `AMD_SMI_MOCK_CONFIG=<path-to-yaml>` - `ROCPROFSYS_AMD_SMI_METRICS="all"` Workload: `./transpose` (mock generates all metric values regardless of workload type). ## Execution ### Phase 1: Setup 1. Auto-discover build directory (`build/debug`, `build/release`, or `build/`) 2. `cd` into build dir, source `share/rocprofiler-systems/setup-env.sh` 3. Verify mock support: `ldd bin/rocprof-sys-run | grep yaml` (yaml-cpp must be linked) 4. Locate mock configs: `~/worktree/rocm-systems/amd-smi-mocks/mock_mi300x.yaml` and `mock_navi48.yaml` 5. Verify `transpose` workload exists ### Phase 2: Execute Test Matrix Run 6 profiling sessions. Use `/tmp/pmc-validate-<run-id>/` output paths. ```bash source share/rocprofiler-systems/setup-env.sh # MI300X — Perfetto Standard AMD_SMI_MOCK_CONFIG=~/worktree/rocm-systems/amd-smi-mocks/mock_mi300x.yaml \ ROCPROFSYS_OUTPUT_PATH=/tmp/pmc-validate-mi300-perfetto \ ROCPROFSYS_TRACE=1 \ ROCPROFSYS_AMD_SMI_METRICS="all" \ ROCPROFSYS_SAMPLING_AINICS="all" \ bin/rocprof-sys-run -- ./transpose # MI300X — Perfetto Legacy AMD_SMI_MOCK_CONFIG=~/worktree/rocm-systems/amd-smi-mocks/mock_mi300x.yaml \ ROCPROFSYS_OUTPUT_PATH=/tmp/pmc-validate-mi300-legacy \ ROCPROFSYS_TRACE=1 \ ROCPROFSYS_TRACE_LEGACY=true \ ROCPROFSYS_AMD_SMI_METRICS="all" \ ROCPROFSYS_SAMPLING_AINICS="all" \ bin/rocprof-sys-run -- ./transpose # MI300X — RocPD AMD_SMI_MOCK_CONFIG=~/worktree/rocm-systems/amd-smi-mocks/mock_mi300x.yaml \ ROCPROFSYS_OUTPUT_PATH=/tmp/pmc-validate-mi300-rocpd \ ROCPROFSYS_USE_ROCPD=1 \ ROCPROFSYS_AMD_SMI_METRICS="all" \ ROCPROFSYS_SAMPLING_AINICS="all" \ bin/rocprof-sys-run -- ./transpose # Navi 48 — Perfetto Standard AMD_SMI_MOCK_CONFIG=~/worktree/rocm-systems/amd-smi-mocks/mock_navi48.yaml \ ROCPROFSYS_OUTPUT_PATH=/tmp/pmc-validate-navi-perfetto \ ROCPROFSYS_TRACE=1 \ ROCPROFSYS_AMD_SMI_METRICS="all" \ bin/rocprof-sys-run -- ./transpose # Navi 48 — Perfetto Legacy AMD_SMI_MOCK_CONFIG=~/worktree/rocm-systems/amd-smi-mocks/mock_navi48.yaml \ ROCPROFSYS_OUTPUT_PATH=/tmp/pmc-validate-navi-legacy \ ROCPROFSYS_TRACE=1 \ ROCPROFSYS_TRACE_LEGACY=true \ ROCPROFSYS_AMD_SMI_METRICS="all" \ bin/rocprof-sys-run -- ./transpose # Navi 48 — RocPD AMD_SMI_MOCK_CONFIG=~/worktree/rocm-systems/amd-smi-mocks/mock_navi48.yaml \ ROCPROFSYS_OUTPUT_PATH=/tmp/pmc-validate-navi-rocpd \ ROCPROFSYS_USE_ROCPD=1 \ ROCPROFSYS_AMD_SMI_METRICS="all" \ bin/rocprof-sys-run -- ./transpose ``` ### Phase 3: Sentinel Detection (CRITICAL) This is the most important validation phase. A single leaked sentinel value is a FAIL. #### Perfetto Traces (Standard and Legacy) Load each trace with MCP tools, then run: ```sql -- SENTINEL CHECK: Find ANY sentinel value in PMC counters SELECT t.name, c.value, c.ts FROM counter c JOIN counter_track t ON c.track_id = t.id WHERE t.name LIKE '%(S)%' AND (c.value = 65535 OR c.value = 4294967295 OR c.value = 18446744073709551615 OR c.value = 1.8446744073709552e+19 OR c.value >= 1.844e+19) LIMIT 20; -- Expected: 0 rows. ANY result is a FAIL. ``` #### RocPD Databases ```bash sqlite3 <db-file> " SELECT p.name, e.value FROM rocpd_pmc_event e JOIN rocpd_info_pmc p ON e.pmc_id = p.id WHERE e.value IN (65535, 4294967295, 18446744073709551615) OR e.value > 1e18 LIMIT 20; " # Expected: 0 rows. ANY result is a FAIL. ``` ### Phase 4: Positive Validation (Configured Metrics Present) Verify that all configured metrics appear in output with expected value ranges. #### Perfetto SQL ```sql -- ALL PMC TRACKS: Get every metric with stats SELECT t.name, COUNT(*) as samples, MIN(c.value) as min_val, MAX(c.value) as max_val, ROUND(AVG(c.value), 2) as avg_val FROM counter c JOIN counter_track t ON c.track_id = t.id WHERE t.name LIKE '%(S)%' GROUP BY t.name ORDER BY t.name; ``` #### RocPD SQL ```bash sqlite3 <db-file> " SELECT p.name, COUNT(*) as samples, MIN(e.value) as min_val, MAX(e.value) as max_val, ROUND(AVG(e.value), 2) as avg_val FROM rocpd_pmc_event e JOIN rocpd_info_pmc p ON e.pmc_id = p.id GROUP BY p.name ORDER BY p.name; " ``` #### MI300X Expected Metrics | Metric | Perfetto Standard Track | Expected Range | Notes |