job-report

# Job Report Generate a factual summary **report card** for an HPC job using GPU telemetry collected by Omnistat. A **single invocation** of `omnistat-inspect ... job <ID> report` returns one JSON document containing every datum needed to render the report. It supports flexible job-context resolution (`--start`/`--end` to skip discovery, `--cache-dir` for cheap repeat calls) and a home for deeper analysis subcommands (e.g. `iterations`). **Target audience:** HPC engineers, AI/ML researchers, system administrators who need a quick, comprehensive snapshot of a job's behavior. **When to use this vs `job-analysis`.** Use **job-report** when you want a quick, factual snapshot of *what* a job did — global statistics, energy, health findings, and data quality, presented as-is. This is NOT an investigation tool: it does not form hypotheses, find root causes, or compare jobs. When you need to understand *why* a job behaved as it did (bottlenecks, throttling, stragglers, regressions, healthy-vs-degraded comparison), use **job-analysis** instead. A common pattern is to run job-report first for the snapshot, then job-analysis if something looks off. ## Bash Tool Description Convention When calling the Bash tool, phrase the `description` field to match the `allowedPrompts` declared in this skill's frontmatter so commands are auto-approved: - `omnistat-inspect` invocation → **"run omnistat-inspect"** - `mktemp` → **"create temporary directory"** - `cat` / reading JSON output files → **"read query results from file"** - Writing the final report → **"write report to file"** ## Prerequisites 1. **Data source** — one of: - **VictoriaMetrics running** with the Omnistat database loaded (use the `open-database` skill if needed), OR - **CSV exports** from `omnistat-query --export` 2. **Python virtual environment activated** with omnistat installed (`pip install ".[query]"` from the omnistat repo root). Confirm `which omnistat-inspect` resolves inside the venv. 3. **Job ID** to report on. ## One-Shot Data Collection `omnistat-inspect ... job <ID> report` writes the JSON document to stdout — redirect it to a file. The data source (`--tsdb-url` / `--csv-dir`) and `--cache-dir` are **global** flags (before `job`); the job ID is a positional argument of the `job` group; `report` is the subcommand. ```bash # 1. Create a scratch directory SCRATCH=$(mktemp -d /tmp/omnistat-report-XXXXXX) echo "Scratch directory: $SCRATCH" # 2a. TSDB mode TSDB_URL="http://localhost:8428" omnistat-inspect --tsdb-url $TSDB_URL job JOBID report > $SCRATCH/report.json # 2b. CSV mode (alternative) omnistat-inspect --csv-dir /path/to/csv/exports job JOBID report > $SCRATCH/report.json ``` Useful flags: | Flag | Position | Purpose | |------|----------|---------| | `--cache-dir DIR` | global (before `job`) | Persist the discovery snapshot and per-module results so repeat calls skip the day-scan and re-computation. | | `--start ISO --end ISO` | `job` group (after JOBID) | Provide the job window directly to **skip job discovery** entirely (much fewer queries). Use with `--interval` when the sampling interval is known. | | `--interval SECONDS` | `job` group | Override discovered sampling interval. | | `--refresh` | `job` group | Force fresh discovery, ignoring any cached snapshot. | | `--cv-threshold 0.05` | `report` subcommand | CV value above which a variance drill-down is reported (default 0.05). | | `--verbose` | `report` subcommand | Include full per-node / per-GPU arrays under `stats.variance.by_node[*].all` and `stats.variance.by_gpu[*].all`, and expand `stats.kernels.top` from the top 10 to **all** `num_kernels` kernels. | That single invocation produces everything the report card needs. ### Job-context flexibility `omnistat-inspect` resolves the job window in one of two ways: - **Discovery (default)** — `omnistat-inspect --tsdb-url $URL job JOBID report` scans the TSDB for the job's time range. Add `--cache-dir $SCRATCH/cache` so a second call (e.g. an `iterations` follow-up) rehydrates from the cached snapshot instead of re-scanning. - **Direct window** — when you already know the job's start/end (e.g. from the scheduler), pass them to skip discovery: ```bash omnistat-inspect --tsdb-url $TSDB_URL job JOBID \ --start 2026-01-01T12:00:00Z --end 2026-01-01T14:30:00Z --interval 10 \ report > $SCRATCH/report.json ``` The `query_stats.total_queries` field will be noticeably lower than a discovery run. (On the TSDB backend the overview still fills hosts/versions live; on CSV the direct path reports only what the window contains.) ## JSON Schema Reference `report.json` is an **envelope** wrapping the report-card payload. The envelope keys are: - `jobid` — the analyzed job id - `generated_at` — ISO 8601 UTC timestamp of when the report was produced - `data_source` — `{type: "tsdb"|"csv", url?, dir?}`; `url` is present for TSDB, `dir` for CSV. This is the source of the "Database" line in Report Metadata. - `overview` — job identity/topology (below) - `stats` — gauges, counters, hardware counters, **and `variance`** (below) - `health` — data-collection coverage **and** health indicators (below) - `query_stats` — `{total_queries, total_query_time_seconds, elapsed_seconds}`; the source of the Report Metadata counts. Note the nesting: `variance` lives under `stats`, and both the data-collection table and the health indicators live under `health`. There is no top-level `variance`, `data_collection`, or `metadata` key. ### `overview` - `jobid`, `user`, `partition` - `start_time`, `end_time` (ISO 8601 UTC) - `duration_seconds`, `duration_human` (e.g. `"2h 34m 12s"`) - `num_nodes`, `num_gpus`, `hosts[]` - `gpu_type`, `driver_version`, `vbios_version` — each is always a sorted list of distinct strings (single-element in the common case), or `null` if the field was not present in the data. - `omnistat_version`, `sampling_interval` (seconds) - `annotations[]` — strings; empty list if none - `figure_of_merit` — list of `{name, instance, min, max, last, num_points}` or `null` ### `stats.gauges[]` Rows in display order. Each entry: `{source, label, name, mean, min, max, unit, n, cv, percentiles}`. `n` is the size of the population behind `cv` / `percentiles` — pooled sample count for plain gauges, per-node-rate count for counter-derived rate gauges (Network RX/TX rate). `min`/`max` are absolute sample extremes over that same population (compare with `stats.variance.by_*[].min`/`max`, which are extremes of *per-key reduced values* — see the `reduction` field there). `percentiles` is `{p5, p25, p50, p75, p95}` over the pooled population. Only metrics that were present in the data are emitted — absent metrics are simply omitted from the list. **Units in the JSON are each metric's native unit** (`W`, `MHz`, `°C`, `%`, `bytes`, `KiB`, `B/s`, `KiB/s`, `J`); labels are unit-free. The renderer is responsible for picking a sensible display unit per row (see "Unit selection" below). ### `stats.counters[]` Each entry: `{source, label, name, total, unit}`. Same base-unit convention as gauges. Same emit-only-if-present rule. ### `stats.hardware_counters` `null` or `{rows: [...], flops: [...] | null, variance: {...}}`. Each `rows[]` entry: `{counter, total, active_rate, effective_rate, observed_span_seconds, monotonic, num_series}`. Counters are summed per GCD (`(instance, card)`) over the full job range using reset-aware `increase()` semantics, so `total` is robust to the ROCm spurious-zero glitch. `active_rate` = `total ÷ observed_span_seconds` (mean per-GCD span actually accumulating); `effective_rate` = `total ÷ job duration` (charges startup/activation idle). `monotonic` is `false` when a downward step (spurious zero, genuine reset, or multiplexing) was detected. Totals/FLOPS assume a **cumulative** counter; **time-multiplexed counters are rate-like (the series churns instead of growing) and are not yet supported**, so an implausible `total`/FLOPS or a `monotonic: false` row likely indicates multiplexing rather than a usable figure. Each `flops[]` entry: `{precision, kind, total_flops, active_rate_flops_per_s, effective_rate_flops_per_s}` (or the whole `flops` is `null`). `active_rate_flops_per_s` divides by the mean per-GCD active span, `effective_rate_flops_per_s` by full wall time. `variance` carries the **per-counter-name spatial variance** of per-GCD counter totals, with the same outer shape as `stats.kernels.variance`: ```jsonc { "cv_threshold": <float>, "metric": "counter_total", // the compared quantity = per-GCD cumulative counter delta "by_node": [ ... ], // key fields: {instance} "by_gpu_id": [ ... ], // key fields: {card} "by_gpu": [ ... ] // key fields: {instance, card} } ``` Each list entry has the same outer shape as a `stats.variance.by_*` entry (`source`/`label`/`name`/`unit`/`reduction`/`n`/`cv`/`min`/`max`/`all`|`percentiles`) **plus a `counter` field** holding the counter name; `source` is `"GPU"`, `label`/`name` are the counter name, `unit` is `"count"`, and `reduction` is `"total"`. The compared quantity is each GCD's **cumulative counter total** — a direct FLOPS-imbalance proxy, since FLOPS scale with the hardware-counter total, so a single straggler GCD surfaces here. All three lists are `[]` when nothing crossed `cv_threshold`. ### `stats.kernels` `null` when kernel tracing was off / no kernel data exists (the report omits the Top kernels table and the folded kernel-dispatch-duration variance rows). Otherwise: ```jsonc { "num_kernels": <int>, // distinct kernels seen across the job "total_dispatches": <int>, // summed over all kernels/GPUs "total_duration_ns": <float>, // summed GPU time over all kernels (ns) "dropped_dispatches": <int>, // node-level dropped count (0 if none) "top": [ // top TOP_KERNELS_LIMIT (10) by default, or all num_kernels kernels under --verbose; ranked by total_duration_ns desc { "kernel": "<full mangled name>", "total_duration_ns": <float>, "dispatches": <int>, "mean_duration_ns": <float> } ], "variance": { "cv_threshold": <float>, "metric": "mean_dispatch_duration_ns", // the compared quantity = Δduration_ns / Δdispatch_count "by_node": [ ... ], // key fields: {instance} "by_gpu_id": [ ... ], // key fields: {card} "by_gpu": [ ... ] // key fields: {instance, card} } } ``` `kernel` names are long mangled C++ symbols (600+ bytes) — keep full names in JSON, **truncate only at render time**. `total_duration_ns` and `mean_duration_ns` are native nanoseconds (the renderer converts). A kernel's share of total GPU time is `total_duration_ns / stats.kernels.total_duration_ns` (compute it at render time — there is no stored percentage field). The three **spatial** variance lists (`by_node`/`by_gpu_id`/`by_gpu`) carry exactly the same outer shape as `stats.variance.by_*` entries (`source`/`label`/`name`/`unit`/`reduction`/`n`/`cv`/`min`/`max`/`all`|`percentiles`) **plus a `kernel` field** holding the full kernel name; `source` is `"GPU"`, `label` is `"Mean dispatch duration"`, `unit` is `"ns"`, and `reduction` is `"ratio"` (Δduration ÷ Δdispatches). The compared quantity is each kernel's **mean dispatch duration** (ns/dispatch); it is a counter ratio rather than a temporal mean, but the outer entry shape matches a gauge entry, so these fold directly into the matching gauge variance subsection. All three variance lists are `[]` when nothing crossed `cv_threshold`. Only the **top kernels** participate in variance (under `--verbose`, "top kernels" is the full kernel set, so variance is computed for every kernel). ### `stats.variance` Nested under `stats` (not a top-level key). - `cv_threshold`, `verbose` - `by_node`, `by_gpu_id`, `by_gpu`: flat lists of per-metric entries (empty list when nothing varied along that axis). Every entry, in every section, carries the same outer shape: ```jsonc { "source": "...", "label": "...", "name": "...", "unit": "...", "reduction": "temporal_mean" | "rate" | "ratio" | "total", "n": <int>, "cv": <float>, "min": {<key_fields>, "value": <float>}, "max": {<key_fields>, "value": <float>}, // exactly one of the following, based on n (both in --verbose): "all": {<key>: <value>, ...} // when n <= 16 "percentiles": {p5, p25, p50, p75, p95} // when n > 16 } ```