job-analysis

# Job Analysis Analyze GPU telemetry data collected by Omnistat for HPC/AI workloads. This skill guides you through a top-down, hypothesis-driven analysis of job performance, driven primarily by the `omnistat-inspect` CLI tool. **Target audience:** HPC engineers, AI/ML researchers, system administrators investigating job performance, GPU health, and resource utilization. **What the analysis produces:** A structured report identifying performance bottlenecks, hardware issues, resource utilization patterns, and anomalies -- with all findings backed by data. **When to use this vs `job-report`.** Use **job-analysis** when you need to understand *why* a job behaved as it did — diagnosing bottlenecks, throttling, stragglers, or regressions, or comparing a degraded job against a healthy baseline. It is hypothesis-driven and iterative. If you only need a quick factual snapshot of *what* a job did (stats, energy, health, data quality) without investigation, use **job-report** instead. A common pattern is to run job-report first, then reach for job-analysis when something looks off. ## Tooling: omnistat-inspect This skill is built entirely around `omnistat-inspect`, the consolidated analysis CLI. It is your single entry point for every phase: - **Baseline characterization** — `omnistat-inspect --tsdb-url $TSDB_URL job JOBID report` produces the structured report card (overview, stats, variance, data-collection, health). Start every analysis here: it is the fastest way to understand scale, runtime, utilization, variance, and health in a single call. - **Deep-dive subcommands** — `job JOBID info` (metadata/topology), `stats` (gauges, counters, hardware counters, and per-node/per-GPU variance), `health` (data-collection coverage and health checks), `iterations` (iteration boundaries and per-iteration stats), `query` (arbitrary PromQL), and `timeseries` (raw series export). - **Data-source inspection** — `omnistat-inspect --tsdb-url $TSDB_URL db info` lists the jobs and metrics available in the backend (no job context required). For anything not covered by a subcommand, drop to raw PromQL via `query` (TSDB) or `curl` against the TSDB HTTP API. ### Job-context flexibility Every `omnistat-inspect job JOBID` invocation resolves the job's time window in one of two ways: - **Discovery (default):** omits `--start`/`--end`; the tool scans the database to discover the job's time range and topology. Add `--cache-dir DIR` to persist the discovery snapshot and per-section results so repeat calls are cheap (no re-scan, no re-query). - **Direct window:** pass both `--start ISO8601` and `--end ISO8601` (optionally `--interval SECONDS`) to skip discovery entirely and analyze an exact window — useful for zooming into a single phase or iteration you found earlier. ```bash # Discovery + cache (cheap repeat calls) omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID report # Direct window (no discovery scan) omnistat-inspect --tsdb-url $TSDB_URL job JOBID \ --start 2026-01-01T12:00:00Z --end 2026-01-01T12:10:00Z report ``` #### Cache-dir reuse for late-pipeline `query` / `timeseries` The `query` and `timeseries` subcommands are typically run late in the analysis, well after discovery. **Always pass the same `--cache-dir` you used for the initial `report`/`info` call** so they rehydrate the cached discovery snapshot instead of re-scanning: ```bash # Early: discovery runs once and is cached (time range + sampling interval) omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID info # Later: query/timeseries reuse the snapshot — no re-scan, correct default step omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID \ query --promql 'avg(rocm_utilization_percentage{$job, $jobstep})' ``` Because the rehydrated snapshot restores the discovered **sampling interval**, the default query step is the sampling interval (`max(sampling_interval, 1s)`) — exactly what you want for full-resolution queries. Notes: - **Do not** reach for `--start`/`--end` just to "scope to the discovered window" — passing them *skips* discovery and drops the sampling interval, so the default step silently degrades to **1s** unless you also add `--interval`. Use `--start`/`--end` only when you genuinely want a narrower sub-window, and pair them with `--interval` to keep the step correct. - For a deliberately coarser step on a single `query` (e.g. an overview of a long job), pass `--step SECONDS` directly; it overrides the default for that call only. (`timeseries` has no `--step`; control its resolution via the cached interval or the global `--interval`.) ## Prerequisites 1. **Data source** — one of: - **VictoriaMetrics running** with the Omnistat database loaded (use the `open-database` skill if needed), OR - **CSV exports** from `omnistat-query --export` (no TSDB required) 2. **Python virtual environment activated** with omnistat installed (`pip install ".[query]"` from the omnistat repo root) — this provides `omnistat-inspect`. Confirm with `which omnistat-inspect`. 3. **Job ID(s)** to analyze (discover available jobs with `omnistat-inspect --tsdb-url $TSDB_URL db info` or `omnistat-inspect --csv-dir /path/to/exports db info`) ## Setup Before starting analysis, set up the working environment: ### TSDB Mode (default) ```bash # 1. Create a cache directory for this analysis session (cheap repeat calls) SCRATCH=$(mktemp -d /tmp/omnistat-inspect-XXXXXX) echo "Cache directory: $SCRATCH/cache" # 2. Set the TSDB URL (VictoriaMetrics or Prometheus) TSDB_URL="http://localhost:8428" # 3. Verify connectivity and discover available jobs omnistat-inspect --tsdb-url $TSDB_URL db info ``` ### CSV Mode Use CSV mode when you have CSV exports from `omnistat-query --export` and no running TSDB. All subcommands except `job query` work in CSV mode — CSV mode uses whatever metrics were exported, so if a metric wasn't included in the export, it won't be available for analysis. ```bash # 1. Create a cache directory for this analysis session SCRATCH=$(mktemp -d /tmp/omnistat-inspect-XXXXXX) echo "Cache directory: $SCRATCH/cache" # 2. Set the CSV directory path CSV_DIR="/path/to/csv/exports" # 3. Discover available data omnistat-inspect --csv-dir $CSV_DIR db info # 4. Run analysis (same subcommands as TSDB mode) omnistat-inspect --csv-dir $CSV_DIR --cache-dir $SCRATCH/cache job JOBID info omnistat-inspect --csv-dir $CSV_DIR --cache-dir $SCRATCH/cache job JOBID stats omnistat-inspect --csv-dir $CSV_DIR --cache-dir $SCRATCH/cache job JOBID health ``` **Note:** The `job query` subcommand (arbitrary PromQL) is not available in CSV mode — it requires a TSDB backend. The `db info` subcommand verifies database connectivity and reports all available jobs with their time ranges, node counts, users, and partitions, plus the full list of available metrics. Use this output to select a job ID and confirm you are looking at the right database. ## Analysis Workflow Follow this top-down, hypothesis-driven workflow. Each phase builds on the previous one. You have freedom to explore and investigate -- this is a methodology guide, not a rigid script. ### Epistemic Discipline **Do not assume what the workload is.** Unless the user tells you the application name, or annotations/metadata explicitly identify it, treat the workload as unknown. Describe what the telemetry shows (e.g., "the GPUs spend ~40% of wall-clock idle between compute phases, each phase ~90s long") rather than what you think it means (e.g., "this is a training workload doing forward/backward passes"). Note: GPU utilization naturally sits near 0% or near 100%, so simply calling it "bimodal" is not an insight — quantify the idle fraction or phase structure instead. If you need to speculate, label it clearly as a hypothesis. **Do not assume the workload is homogeneous.** A single HPC job may run different tasks on different nodes or GPUs. Some nodes may run data loading, others may run compute, others may handle communication. VRAM differences across GPUs, utilization variance across nodes, or non-uniform network traffic are signals of heterogeneity, not necessarily problems. Before reporting "imbalance" as a finding, consider whether the workload is intentionally heterogeneous. **Report what you observe, not what you expect.** If a metric looks unusual, describe the observation and its magnitude. Do not assume it is a problem unless you have evidence of impact (e.g., on runtime, throughput, or health). An observation like "5% of GPUs use 10x more VRAM than the rest" is a fact; "there is a memory imbalance problem" is an interpretation that may be wrong. ### Job Discovery and Characterization **The first step of every analysis is the one-shot report.** It is the factual baseline the rest of the workflow builds on — a single call that returns the job overview, the full `stats` block (gauges, counters, hardware counters, variance), and the `health` block (data-collection coverage + hardware health). Save it and reuse its embedded blocks; the downstream sections below consume this output rather than re-fetching the same data. ```bash # First step — factual baseline. Embeds overview + stats + health in one JSON. omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID report > $SCRATCH/report_JOBID.json # List all metrics available in the data source, plus all jobs and time ranges omnistat-inspect --tsdb-url $TSDB_URL db info ``` The `report` JSON has top-level keys `overview`, `stats`, and `health`. Everything the "Data Collection and Hardware Health Validation" and "Statistical Analysis" sections need is already in this one document — you only issue additional `stats` / `health` / `info` calls to **drill down** (finer grouping), **refresh** (after `--interval` changes), or when you deliberately **skipped** the baseline report. `job info` on its own returns just the overview subset if you ever need it in isolation. Key information to extract: - **Runtime**: How long did the job run? - **Scale**: How many nodes and GPUs? - **Sampling interval**: What time resolution is available? - **Available metrics**: Which collectors were active? (GPU, host, network, RAS, xGMI, rocprofiler) — `db info` lists every metric present in the backend. - **Annotations**: `rmsjob_annotations` markers (e.g., application phases, benchmark identifiers) - **Figure of Merit**: `omnistat_fom` values (e.g., GFLOPS achieved) The `job info` subcommand automatically includes `annotations` and `figure_of_merit` when the corresponding metrics are present in the database. The `job info` subcommand reports the discovered sampling interval (auto-detected from the `omnistat_info` metric's `interval_secs` label). The sampling interval is also used internally by `stats`, `health`, and `iterations` to auto-compute the finest safe query step — you do not need to pass `--interval` to these subcommands. ### GPU Architecture Detection After discovering the job, identify the GPU architecture from the available metrics and load the corresponding architecture profile for GPU-specific domain knowledge (power reporting quirks, thermal limits, memory characteristics, RAS error blocks, hardware counter formulas). Architecture profiles are located in `skills/job-analysis/gpus/`. Read the matching profile before proceeding to data-collection and health validation. **Detection:** Use `overview.gpu_type` from the `job info` output and apply the same substring rules as the job-report skill's "GPU Architecture Handling" table (`MI250` or `MI200 (MCM)` → MI250X; `MI300` → MI300X). When `gpu_type` is a list, apply the rule to each element. The architecture profile contains critical information for correct interpretation of the data (e.g., which GPU cards report power, thermal throttling thresholds, RAS error block meanings). ### Resolution Sensitivity Step resolution significantly affects observed statistics. Coarse steps (e.g., 60s) average over intervals, smearing peaks and troughs together. This can be seriously misleading: - **Peak metrics are underestimated** at coarse resolution (e.g., peak FOM at 60s may be 10-25% lower than at 5s) - **Mean metrics are mostly unaffected** by resolution (averaging preserves the mean) - **Iteration boundaries blur** at coarse resolution, making it impossible to distinguish per-iteration behavior **Always verify critical findings at the finest feasible resolution.** The finest meaningful resolution is the sampling interval reported by `job info` (from `omnistat_info`'s `interval_secs` label) — querying at a finer step than this adds no real data. #### Step Selection The `stats`, `health`, and `iterations` subcommands **auto-compute the finest safe query step**. The step is `max(sampling_interval, runtime / 90000)` — never finer than the actual data, never exceeding VictoriaMetrics' `search.maxPointsPerTimeseries` limit (90,000). There is no arbitrary floor: sub-second sampling intervals are preserved for short jobs where VM limits allow it. `--interval` is a flag on the `job` group and must be placed *before* the subcommand (e.g. `job JOBID --interval N stats`), not after it. The `iterations` subcommand ignores `--interval` — it always uses an auto-computed step. For `stats` and `health` it refines the time range only, while the query step stays auto-computed. For `timeseries` and `query`, the default step is the discovered sampling interval (`max(sampling_interval, 1s)`) when you reuse the cached discovery snapshot via `--cache-dir` — full resolution with no extra flags. For a coarser overview on a long job, `query` accepts an explicit `--step SECONDS`; `timeseries` has no `--step`, so adjust its resolution via the cached interval or the global `--interval`. **When the auto-computed step is much coarser than the sampling interval** (which happens on very long jobs), state the resolution gap explicitly in the report and note which findings may be affected (especially peaks and percentiles). **Critical rule for peak metrics:** If peak FOM, peak utilization, or peak throughput appears degraded, **always re-verify at the finest feasible step** (using `query` with an explicit `--step`) before concluding there is a peak performance difference. Apparent peak degradation is frequently an artifact of temporal averaging — the true peaks may be identical across jobs. Do not claim peak performance differs without checking at fine resolution. ### Data Collection and Hardware Health Validation Before analyzing performance, verify that data collection was complete and reliable, and check for hardware issues. **This data is already in the baseline report's `health` block — read it from there; do not re-fetch.** Only run `health` standalone if you skipped the baseline report or need to refresh after changing `--interval`: ```bash # Standalone health (only if you skipped the baseline report or are refreshing) omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID health ``` The `health` block covers both data-collection coverage (completeness, timing stagger, gaps) and hardware health (RAS errors, thermals, power). #### Data-collection coverage (part of `health`) Review the coverage portion of the health report for: - **Missing nodes**: `expected_nodes` vs `reporting_nodes` — any gap means some nodes never reported - **Activation stagger**: `activation_stagger_seconds` — how long it took for all nodes to start reporting. A spread >5% of total job duration is significant and means early-job statistics are skewed by partial participation - **Deactivation stagger**: `deactivation_stagger_seconds` — same for shutdown. Large spread means late-job statistics are unreliable - **Sampling gaps**: `nodes_with_gaps` and `total_gaps` — gaps are reported only as counts, not per-gap timing. To localize them (e.g., distinguish a clustered systemic event from distributed per-node issues), drill down with `timeseries`/`query` - **Reporting duration**: `reporting_duration_per_node_seconds` (a `{mean, min, max}` object) — nodes with significantly shorter reporting durations may have crashed or been evicted mid-job #### Hardware health (`health`) Review the health report for: - **RAS errors**: Any hardware errors during the job - **Thermal issues**: GPUs running hot - **Power anomalies**: Unexpected zero-power readings - **Push health**: Whether monitoring push duration exceeded the push interval (indicates monitoring overhead) If critical issues are found, note them -- they may explain performance anomalies found later. ### Statistical Analysis Follow these steps in order. **Do not skip steps or move to iteration analysis until all steps are complete.** #### Step 1: Read the baseline stats The job's `stats` are **already in the baseline report** (`report_JOBID.json` → `stats`): global gauge/counter summaries, hardware counters, and per-node / per-GPU variance, all in one block — no `--category` or `--level` flags exist or are needed. Read them from the baseline; only run `stats` standalone if you skipped the report or are refreshing after an `--interval` change: ```bash # Standalone stats (only if you skipped the baseline report or are refreshing) omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID stats > $SCRATCH/stats_JOBID.json ``` The `stats` block has keys `gauges`, `counters`, `hardware_counters`, and `variance`. Counter metrics (cumulative values like bytes transferred, energy consumed) are automatically detected and produce delta-based totals. Gauge metrics produce mean/min/max/cv/percentiles. The `cv` (coefficient of variation) field measures relative dispersion — high CV indicates non-uniform distribution across GPUs or nodes. #### Step 2: Identify anomalous metrics Review the gauge and counter stats. For each metric, check for: - High `cv` (uneven distribution across nodes/GPUs) - Unexpected values (rates, totals, or distributions that differ from expectation) - Large gaps between percentiles (for GPU utilization specifically, a near-0/near-100 split is the expected norm — not a finding; quantify idle time or phase structure rather than labeling it "bimodal") **In comparative analysis:** compare each metric between the healthy and degraded jobs. Identify which show significant differences (>10% in rates or totals, >5 percentage points in gauge means). #### Step 3: Inspect the variance breakdown