job-analysis

name	job-analysis
description	Analyze an HPC job from an Omnistat database using hypothesis-driven exploration, driven by the omnistat-inspect tool. Use this to diagnose why a job behaved as it did — performance bottlenecks, hardware issues, anomalies, or comparing a degraded job against a healthy baseline. For a plain factual snapshot without investigation, use job-report instead.
allowedPrompts	[{"tool":"Bash","prompt":"run omnistat-inspect commands"},{"tool":"Bash","prompt":"execute PromQL queries via curl"},{"tool":"Bash","prompt":"create temporary directory"},{"tool":"Bash","prompt":"read query results from file"},{"tool":"Bash","prompt":"list directory contents"},{"tool":"Bash","prompt":"check victoriametrics status"},{"tool":"Bash","prompt":"run curl commands"}]

Job Analysis

Analyze GPU telemetry data collected by Omnistat for HPC/AI workloads. This skill guides you through a top-down, hypothesis-driven analysis of job performance, driven primarily by the omnistat-inspect CLI tool.

Target audience: HPC engineers, AI/ML researchers, system administrators investigating job performance, GPU health, and resource utilization.

What the analysis produces: A structured report identifying performance bottlenecks, hardware issues, resource utilization patterns, and anomalies -- with all findings backed by data.

When to use this vs job-report. Use job-analysis when you need to understand why a job behaved as it did — diagnosing bottlenecks, throttling, stragglers, or regressions, or comparing a degraded job against a healthy baseline. It is hypothesis-driven and iterative. If you only need a quick factual snapshot of what a job did (stats, energy, health, data quality) without investigation, use job-report instead. A common pattern is to run job-report first, then reach for job-analysis when something looks off.

Tooling: omnistat-inspect

This skill is built entirely around omnistat-inspect, the consolidated analysis CLI. It is your single entry point for every phase:

• Baseline characterization — omnistat-inspect --tsdb-url $TSDB_URL job JOBID report produces the structured report card (overview, stats, variance, data-collection, health). Start every analysis here: it is the fastest way to understand scale, runtime, utilization, variance, and health in a single call.
• Deep-dive subcommands — job JOBID info (metadata/topology), stats (gauges, counters, hardware counters, and per-node/per-GPU variance), health (data-collection coverage and health checks), iterations (iteration boundaries and per-iteration stats), query (arbitrary PromQL), and timeseries (raw series export).
• Data-source inspection — omnistat-inspect --tsdb-url $TSDB_URL db info lists the jobs and metrics available in the backend (no job context required).

For anything not covered by a subcommand, drop to raw PromQL via query (TSDB) or curl against the TSDB HTTP API.

Job-context flexibility

Every omnistat-inspect job JOBID invocation resolves the job's time window in one of two ways:

• Discovery (default): omits --start/--end; the tool scans the database to discover the job's time range and topology. Add --cache-dir DIR to persist the discovery snapshot and per-section results so repeat calls are cheap (no re-scan, no re-query).
• Direct window: pass both --start ISO8601 and --end ISO8601 (optionally --interval SECONDS) to skip discovery entirely and analyze an exact window — useful for zooming into a single phase or iteration you found earlier.

# Discovery + cache (cheap repeat calls)
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID report

# Direct window (no discovery scan)
omnistat-inspect --tsdb-url $TSDB_URL job JOBID \
  --start 2026-01-01T12:00:00Z --end 2026-01-01T12:10:00Z report

Cache-dir reuse for late-pipeline query / timeseries

The query and timeseries subcommands are typically run late in the analysis, well after discovery. Always pass the same --cache-dir you used for the initial report/info call so they rehydrate the cached discovery snapshot instead of re-scanning:

# Early: discovery runs once and is cached (time range + sampling interval)
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID info

# Later: query/timeseries reuse the snapshot — no re-scan, correct default step
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID \
  query --promql 'avg(rocm_utilization_percentage{$job, $jobstep})'

Because the rehydrated snapshot restores the discovered sampling interval, the default query step is the sampling interval (max(sampling_interval, 1s)) — exactly what you want for full-resolution queries. Notes:

• Do not reach for --start/--end just to "scope to the discovered window" — passing them skips discovery and drops the sampling interval, so the default step silently degrades to 1s unless you also add --interval. Use --start/--end only when you genuinely want a narrower sub-window, and pair them with --interval to keep the step correct.
• For a deliberately coarser step on a single query (e.g. an overview of a long job), pass --step SECONDS directly; it overrides the default for that call only. (timeseries has no --step; control its resolution via the cached interval or the global --interval.)

Prerequisites

1. Data source — one of:
   • VictoriaMetrics running with the Omnistat database loaded (use the open-database skill if needed), OR
   • CSV exports from omnistat-query --export (no TSDB required)
2. Python virtual environment activated with omnistat installed (pip install ".[query]" from the omnistat repo root) — this provides omnistat-inspect. Confirm with which omnistat-inspect.
3. Job ID(s) to analyze (discover available jobs with omnistat-inspect --tsdb-url $TSDB_URL db info or omnistat-inspect --csv-dir /path/to/exports db info)

Setup

Before starting analysis, set up the working environment:

TSDB Mode (default)

# 1. Create a cache directory for this analysis session (cheap repeat calls)
SCRATCH=$(mktemp -d /tmp/omnistat-inspect-XXXXXX)
echo "Cache directory: $SCRATCH/cache"

# 2. Set the TSDB URL (VictoriaMetrics or Prometheus)
TSDB_URL="http://localhost:8428"

# 3. Verify connectivity and discover available jobs
omnistat-inspect --tsdb-url $TSDB_URL db info

CSV Mode

Use CSV mode when you have CSV exports from omnistat-query --export and no running TSDB. All subcommands except job query work in CSV mode — CSV mode uses whatever metrics were exported, so if a metric wasn't included in the export, it won't be available for analysis.

# 1. Create a cache directory for this analysis session
SCRATCH=$(mktemp -d /tmp/omnistat-inspect-XXXXXX)
echo "Cache directory: $SCRATCH/cache"

# 2. Set the CSV directory path
CSV_DIR="/path/to/csv/exports"

# 3. Discover available data
omnistat-inspect --csv-dir $CSV_DIR db info

# 4. Run analysis (same subcommands as TSDB mode)
omnistat-inspect --csv-dir $CSV_DIR --cache-dir $SCRATCH/cache job JOBID info
omnistat-inspect --csv-dir $CSV_DIR --cache-dir $SCRATCH/cache job JOBID stats
omnistat-inspect --csv-dir $CSV_DIR --cache-dir $SCRATCH/cache job JOBID health

Note: The job query subcommand (arbitrary PromQL) is not available in CSV mode — it requires a TSDB backend.

The db info subcommand verifies database connectivity and reports all available jobs with their time ranges, node counts, users, and partitions, plus the full list of available metrics. Use this output to select a job ID and confirm you are looking at the right database.

Analysis Workflow

Follow this top-down, hypothesis-driven workflow. Each phase builds on the previous one. You have freedom to explore and investigate -- this is a methodology guide, not a rigid script.

Epistemic Discipline

Do not assume what the workload is. Unless the user tells you the application name, or annotations/metadata explicitly identify it, treat the workload as unknown. Describe what the telemetry shows (e.g., "the GPUs spend ~40% of wall-clock idle between compute phases, each phase ~90s long") rather than what you think it means (e.g., "this is a training workload doing forward/backward passes"). Note: GPU utilization naturally sits near 0% or near 100%, so simply calling it "bimodal" is not an insight — quantify the idle fraction or phase structure instead. If you need to speculate, label it clearly as a hypothesis.

Do not assume the workload is homogeneous. A single HPC job may run different tasks on different nodes or GPUs. Some nodes may run data loading, others may run compute, others may handle communication. VRAM differences across GPUs, utilization variance across nodes, or non-uniform network traffic are signals of heterogeneity, not necessarily problems. Before reporting "imbalance" as a finding, consider whether the workload is intentionally heterogeneous.

Report what you observe, not what you expect. If a metric looks unusual, describe the observation and its magnitude. Do not assume it is a problem unless you have evidence of impact (e.g., on runtime, throughput, or health). An observation like "5% of GPUs use 10x more VRAM than the rest" is a fact; "there is a memory imbalance problem" is an interpretation that may be wrong.

Job Discovery and Characterization

The first step of every analysis is the one-shot report. It is the factual baseline the rest of the workflow builds on — a single call that returns the job overview, the full stats block (gauges, counters, hardware counters, variance), and the health block (data-collection coverage + hardware health). Save it and reuse its embedded blocks; the downstream sections below consume this output rather than re-fetching the same data.

# First step — factual baseline. Embeds overview + stats + health in one JSON.
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID report > $SCRATCH/report_JOBID.json

# List all metrics available in the data source, plus all jobs and time ranges
omnistat-inspect --tsdb-url $TSDB_URL db info

The report JSON has top-level keys overview, stats, and health. Everything the "Data Collection and Hardware Health Validation" and "Statistical Analysis" sections need is already in this one document — you only issue additional stats / health / info calls to drill down (finer grouping), refresh (after --interval changes), or when you deliberately skipped the baseline report. job info on its own returns just the overview subset if you ever need it in isolation.

Key information to extract:

• Runtime: How long did the job run?
• Scale: How many nodes and GPUs?
• Sampling interval: What time resolution is available?
• Available metrics: Which collectors were active? (GPU, host, network, RAS, xGMI, rocprofiler) — db info lists every metric present in the backend.
• Annotations: rmsjob_annotations markers (e.g., application phases, benchmark identifiers)
• Figure of Merit: omnistat_fom values (e.g., GFLOPS achieved)

The job info subcommand automatically includes annotations and figure_of_merit when the corresponding metrics are present in the database.

The job info subcommand reports the discovered sampling interval (auto-detected from the omnistat_info metric's interval_secs label). The sampling interval is also used internally by stats, health, and iterations to auto-compute the finest safe query step — you do not need to pass --interval to these subcommands.

GPU Architecture Detection

After discovering the job, identify the GPU architecture from the available metrics and load the corresponding architecture profile for GPU-specific domain knowledge (power reporting quirks, thermal limits, memory characteristics, RAS error blocks, hardware counter formulas).

Architecture profiles are located in skills/job-analysis/gpus/. Read the matching profile before proceeding to data-collection and health validation.

Detection: Use overview.gpu_type from the job info output and apply the same substring rules as the job-report skill's "GPU Architecture Handling" table (MI250 or MI200 (MCM) → MI250X; MI300 → MI300X). When gpu_type is a list, apply the rule to each element.

The architecture profile contains critical information for correct interpretation of the data (e.g., which GPU cards report power, thermal throttling thresholds, RAS error block meanings).

Resolution Sensitivity

Step resolution significantly affects observed statistics. Coarse steps (e.g., 60s) average over intervals, smearing peaks and troughs together. This can be seriously misleading:

• Peak metrics are underestimated at coarse resolution (e.g., peak FOM at 60s may be 10-25% lower than at 5s)
• Mean metrics are mostly unaffected by resolution (averaging preserves the mean)
• Iteration boundaries blur at coarse resolution, making it impossible to distinguish per-iteration behavior

Always verify critical findings at the finest feasible resolution. The finest meaningful resolution is the sampling interval reported by job info (from omnistat_info's interval_secs label) — querying at a finer step than this adds no real data.

Step Selection

The stats, health, and iterations subcommands auto-compute the finest safe query step. The step is max(sampling_interval, runtime / 90000) — never finer than the actual data, never exceeding VictoriaMetrics' search.maxPointsPerTimeseries limit (90,000). There is no arbitrary floor: sub-second sampling intervals are preserved for short jobs where VM limits allow it.

--interval is a flag on the job group and must be placed before the subcommand (e.g. job JOBID --interval N stats), not after it. The iterations subcommand ignores --interval — it always uses an auto-computed step. For stats and health it refines the time range only, while the query step stays auto-computed.

For timeseries and query, the default step is the discovered sampling interval (max(sampling_interval, 1s)) when you reuse the cached discovery snapshot via --cache-dir — full resolution with no extra flags. For a coarser overview on a long job, query accepts an explicit --step SECONDS; timeseries has no --step, so adjust its resolution via the cached interval or the global --interval.

When the auto-computed step is much coarser than the sampling interval (which happens on very long jobs), state the resolution gap explicitly in the report and note which findings may be affected (especially peaks and percentiles).

Critical rule for peak metrics: If peak FOM, peak utilization, or peak throughput appears degraded, always re-verify at the finest feasible step (using query with an explicit --step) before concluding there is a peak performance difference. Apparent peak degradation is frequently an artifact of temporal averaging — the true peaks may be identical across jobs. Do not claim peak performance differs without checking at fine resolution.

Data Collection and Hardware Health Validation

Before analyzing performance, verify that data collection was complete and reliable, and check for hardware issues. This data is already in the baseline report's health block — read it from there; do not re-fetch. Only run health standalone if you skipped the baseline report or need to refresh after changing --interval:

# Standalone health (only if you skipped the baseline report or are refreshing)
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID health

The health block covers both data-collection coverage (completeness, timing stagger, gaps) and hardware health (RAS errors, thermals, power).

Data-collection coverage (part of health)

Review the coverage portion of the health report for:

• Missing nodes: expected_nodes vs reporting_nodes — any gap means some nodes never reported
• Activation stagger: activation_stagger_seconds — how long it took for all nodes to start reporting. A spread >5% of total job duration is significant and means early-job statistics are skewed by partial participation
• Deactivation stagger: deactivation_stagger_seconds — same for shutdown. Large spread means late-job statistics are unreliable
• Sampling gaps: nodes_with_gaps and total_gaps — gaps are reported only as counts, not per-gap timing. To localize them (e.g., distinguish a clustered systemic event from distributed per-node issues), drill down with timeseries/query
• Reporting duration: reporting_duration_per_node_seconds (a {mean, min, max} object) — nodes with significantly shorter reporting durations may have crashed or been evicted mid-job

Hardware health (health)

Review the health report for:

• RAS errors: Any hardware errors during the job
• Thermal issues: GPUs running hot
• Power anomalies: Unexpected zero-power readings
• Push health: Whether monitoring push duration exceeded the push interval (indicates monitoring overhead)

If critical issues are found, note them -- they may explain performance anomalies found later.

Statistical Analysis

Follow these steps in order. Do not skip steps or move to iteration analysis until all steps are complete.

Step 1: Read the baseline stats

The job's stats are already in the baseline report (report_JOBID.json → stats): global gauge/counter summaries, hardware counters, and per-node / per-GPU variance, all in one block — no --category or --level flags exist or are needed. Read them from the baseline; only run stats standalone if you skipped the report or are refreshing after an --interval change:

# Standalone stats (only if you skipped the baseline report or are refreshing)
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID stats > $SCRATCH/stats_JOBID.json

The stats block has keys gauges, counters, hardware_counters, and variance. Counter metrics (cumulative values like bytes transferred, energy consumed) are automatically detected and produce delta-based totals. Gauge metrics produce mean/min/max/cv/percentiles. The cv (coefficient of variation) field measures relative dispersion — high CV indicates non-uniform distribution across GPUs or nodes.

Step 2: Identify anomalous metrics

Review the gauge and counter stats. For each metric, check for:

• High cv (uneven distribution across nodes/GPUs)
• Unexpected values (rates, totals, or distributions that differ from expectation)
• Large gaps between percentiles (for GPU utilization specifically, a near-0/near-100 split is the expected norm — not a finding; quantify idle time or phase structure rather than labeling it "bimodal")

In comparative analysis: compare each metric between the healthy and degraded jobs. Identify which show significant differences (>10% in rates or totals, >5 percentage points in gauge means).

Step 3: Inspect the variance breakdown