| name | verify-pmc-metrics |
| description | Mock-based rigorous PMC metric validation — tests sentinel filtering, value ranges, and architecture-specific behavior across Perfetto and RocPD |
Mock-Based PMC Validation
Deterministic validation of AMD SMI Performance Monitoring Counter (PMC) metrics using the mock system (AMD_SMI_MOCK_CONFIG). Tests sentinel filtering, expected value ranges, positive presence of configured metrics, and negative absence of unsupported metrics across all output formats.
- Execute all profiling and validation automatically — do NOT ask for user confirmation.
- Run ALL 6 test matrix entries (2 architectures x 3 formats).
- Sentinel detection is THE critical check — a single leaked sentinel is a FAIL.
- Use rocprof-sys MCP tools for Perfetto traces, sqlite3 for RocPD databases.
- Mock configs are in the worktree at `~/worktree/rocm-systems/amd-smi-mocks/`.
Sentinel Value Reference
GPU firmware returns these values for unsupported/unavailable metrics. They must NEVER appear in output.
| Type | Hex | Decimal | Float (Perfetto) | Where Used |
|---|
uint16_t | 0xFFFF | 65535 | 65535.0 | Temperature, activity %, power, voltage, clocks, PCIe width/speed, VCN/JPEG arrays |
uint32_t | 0xFFFFFFFF | 4294967295 | 4294967295.0 | Various 32-bit fields |
uint64_t | 0xFFFFFFFFFFFFFFFF | 18446744073709551615 | 1.8446744073709552e+19 | PCIe bandwidth, XGMI data, memory |
Perfetto stores counter values as doubles, so UINT64_MAX appears as 1.8446744073709552e+19. Both exact and float representations must be checked.
Known Sentinel Filtering Pipeline
| Metric | device.hpp | perfetto_policy.hpp | perfetto_processor.cpp | rocpd_processor.cpp |
|---|
| VCN/JPEG (uint16) | Support check only | Filtered | Filtered | NOT filtered |
| XGMI (uint64) | Zeroed via populate_if_supported() | N/A | Double-checked | Filtered |
| PCIe (uint16/64) | Zeroed via populate_if_supported() | N/A | Relies on zero check | NOT filtered |
| NIC RDMA | N/A | N/A | No filtering | No filtering |
Known leakage risks:
- RocPD VCN/JPEG array values (no explicit sentinel filter)
- RocPD PCIe values (relies on device.hpp zeroing only)
- NIC metrics in all outputs (no sentinel filtering anywhere)
Mock Configurations
Two architectures test complementary metric subsets. Located in ~/worktree/rocm-systems/amd-smi-mocks/.
MI300X (mock_mi300x.yaml)
Data-center GPU with per-XCP metrics, HBM, and NIC.
| Category | What's Configured | Expected Range | Notes |
|---|
| Temperature | hotspot only | [55, 75] | No temperature_edge (sentineled) |
| HBM Temperature | 4 active / 4 total | [40, 60] | Array: active entries have values, rest sentineled |
| Power | current_socket_power | [300, 500] | Watts |
| GFX Busy | gfx_activity | [80, 100] | % |
| UMC Busy | umc_activity | [0, 10] | % |
| MM Busy | NOT configured | N/A | Should be absent (sentineled) |
| VCN Busy | per-XCP, 4 active engines | [0, 10] | Per-XCP format, NOT device-level VCN Activity |
| JPEG Busy | per-XCP, 32 active engines | [0, 10] | Per-XCP format, NOT device-level JPEG Activity |
| PCIe Width | Fixed 16 | 16 | Lanes |
| PCIe Speed | Fixed 32000 | 32000 | MT/s |
| XGMI | NOT configured | N/A | Should be absent (sentineled) |
| SDMA | 2 processes | varies | Cumulative byte counters |
| Memory | vram_used | varies | Derived from vram_used / vram_total |
| Voltages | NOT configured | N/A | Should be absent (sentineled) |
| Fan | NOT configured | N/A | Should be absent (sentineled) |
| NIC RDMA | 6 stats configured | [0, 1000000] | RX/TX bytes, packets, CNP |
Navi 48 (mock_navi48.yaml)
Consumer GPU with device-level metrics, no XCP, no NIC.
| Category | What's Configured | Expected Range | Notes |
|---|
| Temperature | edge, hotspot, mem | [35, 75] | Has temperature_edge |
| HBM Temperature | NOT configured | N/A | Should be absent (sentineled) |
| Power | current_socket_power | [20, 30] | Watts |
| GFX Busy | gfx_activity | [0, 10] | % |
| UMC Busy | umc_activity | [0, 5] | % |
| MM Busy | mm_activity | [0, 10] | % (present, unlike MI300X) |
| VCN Activity | device-level, 1 active / 4 total | [0, 10] | Device-level, NOT per-XCP |
| JPEG Activity | NOT configured | N/A | Should be absent (sentineled) |
| PCIe Width | Fixed 16 | 16 | Lanes |
| PCIe Speed | Fixed 2500 | 2500 | MT/s |
| XGMI | NOT configured | N/A | Should be absent |
| SDMA | 1 process | varies | Cumulative byte counters |
| Memory | vram_used | varies | Derived |
| Voltages | gfx, soc, mem configured | [700, 1200] | mV |
| Fan | rpm and max configured | [800, 1200] | RPM |
| NIC | No NIC devices | N/A | No NIC tracks should appear |
Output Formats
| # | Format | Environment Variables |
|---|
| 1 | Perfetto Standard | ROCPROFSYS_TRACE=1 |
| 2 | Perfetto Legacy | ROCPROFSYS_TRACE=1 ROCPROFSYS_TRACE_LEGACY=true |
| 3 | RocPD | ROCPROFSYS_USE_ROCPD=1 |
Perfetto Track Name Differences
| Format | GPU Pattern | NIC Pattern |
|---|
| Standard | GPU [N] <Metric> (S) | NIC [N] <Metric> (S) |
| Legacy | GPU <Metric> [N] (S) | NIC <device_name> <Metric> [N] (S) |
Legacy also uses abbreviated forms: Pkts vs Packets.
Test Matrix
| Run ID | Config | Format | Extra Env |
|---|
| mi300-perfetto | mock_mi300x.yaml | Perfetto Standard | ROCPROFSYS_SAMPLING_AINICS="all" |
| mi300-legacy | mock_mi300x.yaml | Perfetto Legacy | ROCPROFSYS_SAMPLING_AINICS="all" |
| mi300-rocpd | mock_mi300x.yaml | RocPD | ROCPROFSYS_SAMPLING_AINICS="all" |
| navi-perfetto | mock_navi48.yaml | Perfetto Standard | — |
| navi-legacy | mock_navi48.yaml | Perfetto Legacy | — |
| navi-rocpd | mock_navi48.yaml | RocPD | — |
Common env for ALL runs:
AMD_SMI_MOCK_CONFIG=<path-to-yaml>ROCPROFSYS_AMD_SMI_METRICS="all"
Workload: ./transpose (mock generates all metric values regardless of workload type).
Execution
Phase 1: Setup
- Auto-discover build directory (
build/debug, build/release, or build/)
cd into build dir, source share/rocprofiler-systems/setup-env.sh- Verify mock support:
ldd bin/rocprof-sys-run | grep yaml (yaml-cpp must be linked)
- Locate mock configs:
~/worktree/rocm-systems/amd-smi-mocks/mock_mi300x.yaml and mock_navi48.yaml
- Verify
transpose workload exists
Phase 2: Execute Test Matrix
Run 6 profiling sessions. Use /tmp/pmc-validate-<run-id>/ output paths.
source share/rocprofiler-systems/setup-env.sh
AMD_SMI_MOCK_CONFIG=~/worktree/rocm-systems/amd-smi-mocks/mock_mi300x.yaml \
ROCPROFSYS_OUTPUT_PATH=/tmp/pmc-validate-mi300-perfetto \
ROCPROFSYS_TRACE=1 \
ROCPROFSYS_AMD_SMI_METRICS="all" \
ROCPROFSYS_SAMPLING_AINICS="all" \
bin/rocprof-sys-run -- ./transpose
AMD_SMI_MOCK_CONFIG=~/worktree/rocm-systems/amd-smi-mocks/mock_mi300x.yaml \
ROCPROFSYS_OUTPUT_PATH=/tmp/pmc-validate-mi300-legacy \
ROCPROFSYS_TRACE=1 \
ROCPROFSYS_TRACE_LEGACY=true \
ROCPROFSYS_AMD_SMI_METRICS="all" \
ROCPROFSYS_SAMPLING_AINICS="all" \
bin/rocprof-sys-run -- ./transpose
AMD_SMI_MOCK_CONFIG=~/worktree/rocm-systems/amd-smi-mocks/mock_mi300x.yaml \
ROCPROFSYS_OUTPUT_PATH=/tmp/pmc-validate-mi300-rocpd \
ROCPROFSYS_USE_ROCPD=1 \
ROCPROFSYS_AMD_SMI_METRICS="all" \
ROCPROFSYS_SAMPLING_AINICS="all" \
bin/rocprof-sys-run -- ./transpose
AMD_SMI_MOCK_CONFIG=~/worktree/rocm-systems/amd-smi-mocks/mock_navi48.yaml \
ROCPROFSYS_OUTPUT_PATH=/tmp/pmc-validate-navi-perfetto \
ROCPROFSYS_TRACE=1 \
ROCPROFSYS_AMD_SMI_METRICS="all" \
bin/rocprof-sys-run -- ./transpose
AMD_SMI_MOCK_CONFIG=~/worktree/rocm-systems/amd-smi-mocks/mock_navi48.yaml \
ROCPROFSYS_OUTPUT_PATH=/tmp/pmc-validate-navi-legacy \
ROCPROFSYS_TRACE=1 \
ROCPROFSYS_TRACE_LEGACY=true \
ROCPROFSYS_AMD_SMI_METRICS="all" \
bin/rocprof-sys-run -- ./transpose
AMD_SMI_MOCK_CONFIG=~/worktree/rocm-systems/amd-smi-mocks/mock_navi48.yaml \
ROCPROFSYS_OUTPUT_PATH=/tmp/pmc-validate-navi-rocpd \
ROCPROFSYS_USE_ROCPD=1 \
ROCPROFSYS_AMD_SMI_METRICS="all" \
bin/rocprof-sys-run -- ./transpose
Phase 3: Sentinel Detection (CRITICAL)
This is the most important validation phase. A single leaked sentinel value is a FAIL.
Perfetto Traces (Standard and Legacy)
Load each trace with MCP tools, then run:
SELECT t.name, c.value, c.ts
FROM counter c JOIN counter_track t ON c.track_id = t.id
WHERE t.name LIKE '%(S)%'
AND (c.value = 65535
OR c.value = 4294967295
OR c.value = 18446744073709551615
OR c.value = 1.8446744073709552e+19
OR c.value >= 1.844e+19)
LIMIT 20;
RocPD Databases
sqlite3 <db-file> "
SELECT p.name, e.value
FROM rocpd_pmc_event e JOIN rocpd_info_pmc p ON e.pmc_id = p.id
WHERE e.value IN (65535, 4294967295, 18446744073709551615)
OR e.value > 1e18
LIMIT 20;
"
Phase 4: Positive Validation (Configured Metrics Present)
Verify that all configured metrics appear in output with expected value ranges.
Perfetto SQL
SELECT t.name,
COUNT(*) as samples,
MIN(c.value) as min_val,
MAX(c.value) as max_val,
ROUND(AVG(c.value), 2) as avg_val
FROM counter c JOIN counter_track t ON c.track_id = t.id
WHERE t.name LIKE '%(S)%'
GROUP BY t.name
ORDER BY t.name;
RocPD SQL
sqlite3 <db-file> "
SELECT p.name, COUNT(*) as samples,
MIN(e.value) as min_val, MAX(e.value) as max_val,
ROUND(AVG(e.value), 2) as avg_val
FROM rocpd_pmc_event e JOIN rocpd_info_pmc p ON e.pmc_id = p.id
GROUP BY p.name ORDER BY p.name;
"
MI300X Expected Metrics
| Metric | Perfetto Standard Track | Expected Range | Notes |
|---|
| Temperature (hotspot) | GPU [0] Temperature (S) | [55, 75] | Celsius |
| Power | GPU [0] Current Power (S) | [300, 500] | Watts |
| GFX Busy | GPU [0] GFX Busy (S) | [80, 100] | % |
| UMC Busy | GPU [0] UMC Busy (S) | [0, 10] | % |
| Memory Usage | GPU [0] Memory Usage (S) | >0 | MB, derived from vram |
| VCN Busy | GPU [0] VCN Busy ... (S) | [0, 10] | Per-XCP, 4 active engines |
| JPEG Busy | GPU [0] JPEG Busy ... (S) | [0, 10] | Per-XCP, 32 active engines |
| PCIe Width | GPU [0] PCIe Link Width (S) | 16 | Fixed |
| PCIe Speed | GPU [0] PCIe Link Speed (S) | 32000 | Fixed MT/s |
| SDMA Usage | GPU [0] SDMA Usage (S) | >=0 | Derived from cumulative counters |
| NIC RX Bytes | NIC [0] RX RDMA Bytes (S) | [0, 1000000] | |
| NIC TX Bytes | NIC [0] TX RDMA Bytes (S) | [0, 1000000] | |
| NIC RX Packets | NIC [0] RX RDMA Packets (S) | [0, 1000000] | |
| NIC TX Packets | NIC [0] TX RDMA Packets (S) | [0, 1000000] | |
| NIC RX CNP | NIC [0] RX CNP Packets (S) | [0, 1000000] | |
| NIC TX CNP | NIC [0] TX CNP Packets (S) | [0, 1000000] | |
Navi 48 Expected Metrics
| Metric | Perfetto Standard Track | Expected Range | Notes |
|---|
| Temperature (edge) | GPU [0] Temperature (S) | [35, 55] | Has edge temp |
| Power | GPU [0] Current Power (S) | [20, 30] | Watts |
| GFX Busy | GPU [0] GFX Busy (S) | [0, 10] | % |
| UMC Busy | GPU [0] UMC Busy (S) | [0, 5] | % |
| MM Busy | GPU [0] MM Busy (S) | [0, 10] | % (absent on MI300X) |
| VCN Activity | GPU [0] VCN Activity ... (S) | [0, 10] | Device-level, 1 active |
| Memory Usage | GPU [0] Memory Usage (S) | >0 | MB |
| PCIe Width | GPU [0] PCIe Link Width (S) | 16 | Fixed |
| PCIe Speed | GPU [0] PCIe Link Speed (S) | 2500 | Fixed MT/s |
| SDMA Usage | GPU [0] SDMA Usage (S) | >=0 | 1 process |
Range Validation SQL
For each metric with a known range, verify no values fall outside:
SELECT t.name, c.value
FROM counter c JOIN counter_track t ON c.track_id = t.id
WHERE t.name LIKE '%Current Power%' AND t.name LIKE '%(S)%'
AND (c.value < 300 OR c.value > 500)
LIMIT 10;
Repeat for each metric with known range bounds from the mock config.
Phase 5: Negative Validation (Unsupported Metrics Absent)
Verify that metrics NOT configured in the mock do NOT appear in output.
MI300X Should NOT Have
| Absent Metric | Perfetto Pattern | RocPD Pattern |
|---|
| Temperature Edge | %Temperature Edge% or %temperature_edge% | device_temp_edge |
| MM Busy | %MM Busy% or %mm_activity% | device_busy_mm |
| VCN Activity (device-level) | %VCN Activity% | device_vcn_activity |
| JPEG Activity (device-level) | %JPEG Activity% | device_jpeg_activity |
| XGMI | %XGMI% | device_xgmi% |
| Voltage | %Voltage% | device_voltage% |
| Fan | %Fan% | device_fan% |
Navi 48 Should NOT Have
| Absent Metric | Perfetto Pattern | RocPD Pattern |
|---|
| HBM Temperature | %HBM Temp% | device_temp_hbm% |
| VCN Busy (per-XCP) | %VCN Busy% | N/A |
| JPEG Busy (per-XCP) | %JPEG Busy% | N/A |
| JPEG Activity | %JPEG Activity% | device_jpeg_activity% |
| XGMI | %XGMI% | device_xgmi% |
| NIC | %NIC% | nic_% |
Perfetto Negative Check SQL
SELECT t.name FROM counter_track t
WHERE t.name LIKE '%(S)%'
AND (t.name LIKE '%Temperature Edge%'
OR t.name LIKE '%MM Busy%'
OR t.name LIKE '%VCN Activity%'
OR t.name LIKE '%JPEG Activity%'
OR t.name LIKE '%XGMI%'
OR t.name LIKE '%Voltage%'
OR t.name LIKE '%Fan%');
SELECT t.name FROM counter_track t
WHERE t.name LIKE '%(S)%'
AND (t.name LIKE '%HBM Temp%'
OR t.name LIKE '%VCN Busy%'
OR t.name LIKE '%JPEG Busy%'
OR t.name LIKE '%JPEG Activity%'
OR t.name LIKE '%XGMI%'
OR t.name LIKE '%NIC%');
RocPD Negative Check
sqlite3 <db-file> "
SELECT DISTINCT p.name FROM rocpd_info_pmc p
WHERE p.name IN ('device_busy_mm', 'device_temp_edge')
OR p.name LIKE 'device_vcn_activity%'
OR p.name LIKE 'device_jpeg_activity%'
OR p.name LIKE 'device_xgmi%'
OR p.name LIKE 'device_voltage%'
OR p.name LIKE 'device_fan%';
"
sqlite3 <db-file> "
SELECT DISTINCT p.name FROM rocpd_info_pmc p
WHERE p.name LIKE 'device_temp_hbm%'
OR p.name LIKE 'device_jpeg_activity%'
OR p.name LIKE 'device_xgmi%'
OR p.name LIKE 'nic_%';
"
Phase 6: Cross-Format Consistency
For each architecture, verify that the same set of metrics appears across all 3 formats (accounting for naming differences):
SELECT COUNT(DISTINCT t.name) FROM counter_track t WHERE t.name LIKE '%(S)%';
For RocPD, compare the set of distinct rocpd_info_pmc.name values against Perfetto track names.
Discrepancies between formats indicate a bug in one of the output processors.
Phase 7: Cleanup
rm -rf /tmp/pmc-validate-mi300-perfetto /tmp/pmc-validate-mi300-legacy /tmp/pmc-validate-mi300-rocpd
rm -rf /tmp/pmc-validate-navi-perfetto /tmp/pmc-validate-navi-legacy /tmp/pmc-validate-navi-rocpd
Report Format
# PMC Validation Report (Mock-Based)
**Date**: YYYY-MM-DD
**Build**: <build_dir>
**Branch**: <branch>
## Sentinel Check Results
| Config | Format | Sentinel Values Found | Status |
|--------|--------|--------------------|--------|
| MI300X | Perfetto | 0 | PASS |
| MI300X | Legacy | 0 | PASS |
| MI300X | RocPD | 0 | PASS |
| Navi 48 | Perfetto | 0 | PASS |
| Navi 48 | Legacy | 0 | PASS |
| Navi 48 | RocPD | 0 | PASS |
## Positive Validation (Configured Metrics)
| Config | Metric | Perfetto | Legacy | RocPD | Range OK |
|--------|--------|----------|--------|-------|----------|
| MI300X | Power | PASS (N, [300-500]) | PASS | PASS | YES |
| MI300X | Temperature | PASS (N, [55-75]) | PASS | PASS | YES |
| ... | ... | ... | ... | ... | ... |
| Navi 48 | Power | PASS (N, [20-30]) | PASS | PASS | YES |
| ... | ... | ... | ... | ... | ... |
## Negative Validation (Unsupported Metrics)
| Config | Metric Should Be Absent | Perfetto | Legacy | RocPD |
|--------|------------------------|----------|--------|-------|
| MI300X | temperature_edge | PASS (absent) | PASS | PASS |
| MI300X | MM Busy | PASS (absent) | PASS | PASS |
| MI300X | XGMI | PASS (absent) | PASS | PASS |
| Navi 48 | HBM Temperature | PASS (absent) | PASS | PASS |
| Navi 48 | JPEG Activity | PASS (absent) | PASS | PASS |
| Navi 48 | NIC | PASS (absent) | PASS | PASS |
| ... | ... | ... | ... | ... |
## Cross-Format Consistency
| Config | Perfetto Metrics | Legacy Metrics | RocPD Metrics | Match |
|--------|-----------------|----------------|---------------|-------|
| MI300X | N | N | N | YES |
| Navi 48 | N | N | N | YES |
## Issues Found
[List with severity: CRITICAL / HIGH / MEDIUM / LOW]
## Conclusion
[Overall assessment]
Save report to: planning/pmc-validation-report.md in the project worktree.
Common Issues & Known Limitations
| Issue | Description | Impact |
|---|
| Perfetto double precision | UINT64_MAX stored as 1.8446744073709552e+19 | Must check >= 1.844e+19 in addition to exact values |
| Legacy timing | Short workloads (<0.5s) may get 0-1 samples in Legacy format | Use transpose which runs ~2.4s |
| SDMA cumulative | SDMA values are cumulative byte counters, not instantaneous rates | Delta calculation needed for rate validation |
| Mock workload independence | Mock generates values regardless of workload type | VCN/JPEG don't need decode binaries |
| yaml-cpp dependency | Mock requires yaml-cpp linked into the build | Check with ldd before running |
| Invalid timestamps (Legacy) | First sample may have invalid timestamp, dropped with warning | Reduces sample count by 1-2 |
Integration
| Outcome | Next Step |
|---|
| Sentinel found | planning-bugfix — plan fix for the leaking pipeline stage |
| Missing configured metric | planning-bugfix — investigate why metric isn't reaching output |
| Unsupported metric present | planning-bugfix — sentinel filtering gap |
| All checks pass | git-prepare-pull-request to submit |
This skill is also agent-level knowledge for the orchestrator's Tester agent.
