job-report

name	job-report
description	Generate a factual summary report card for an HPC job from an Omnistat database using the single-shot omnistat-inspect JSON command. Use this for a quick, comprehensive snapshot of what a job did (stats, energy, health, data quality) without diagnosing why. For root-cause investigation, performance debugging, or comparing jobs, use job-analysis instead.
allowedPrompts	[{"tool":"Bash","prompt":"run omnistat-inspect"},{"tool":"Bash","prompt":"create temporary directory"},{"tool":"Bash","prompt":"read query results from file"},{"tool":"Bash","prompt":"write report to file"}]

Job Report

Generate a factual summary report card for an HPC job using GPU telemetry collected by Omnistat. A single invocation of omnistat-inspect ... job <ID> report returns one JSON document containing every datum needed to render the report. It supports flexible job-context resolution (--start/--end to skip discovery, --cache-dir for cheap repeat calls) and a home for deeper analysis subcommands (e.g. iterations).

Target audience: HPC engineers, AI/ML researchers, system administrators who need a quick, comprehensive snapshot of a job's behavior.

When to use this vs job-analysis. Use job-report when you want a quick, factual snapshot of what a job did — global statistics, energy, health findings, and data quality, presented as-is. This is NOT an investigation tool: it does not form hypotheses, find root causes, or compare jobs. When you need to understand why a job behaved as it did (bottlenecks, throttling, stragglers, regressions, healthy-vs-degraded comparison), use job-analysis instead. A common pattern is to run job-report first for the snapshot, then job-analysis if something looks off.

Bash Tool Description Convention

When calling the Bash tool, phrase the description field to match the allowedPrompts declared in this skill's frontmatter so commands are auto-approved:

• omnistat-inspect invocation → "run omnistat-inspect"
• mktemp → "create temporary directory"
• cat / reading JSON output files → "read query results from file"
• Writing the final report → "write report to file"

Prerequisites

1. Data source — one of:
   • VictoriaMetrics running with the Omnistat database loaded (use the open-database skill if needed), OR
   • CSV exports from omnistat-query --export
2. Python virtual environment activated with omnistat installed (pip install ".[query]" from the omnistat repo root). Confirm which omnistat-inspect resolves inside the venv.
3. Job ID to report on.

One-Shot Data Collection

omnistat-inspect ... job <ID> report writes the JSON document to stdout — redirect it to a file. The data source (--tsdb-url / --csv-dir) and --cache-dir are global flags (before job); the job ID is a positional argument of the job group; report is the subcommand.

# 1. Create a scratch directory
SCRATCH=$(mktemp -d /tmp/omnistat-report-XXXXXX)
echo "Scratch directory: $SCRATCH"

# 2a. TSDB mode
TSDB_URL="http://localhost:8428"
omnistat-inspect --tsdb-url $TSDB_URL job JOBID report > $SCRATCH/report.json

# 2b. CSV mode (alternative)
omnistat-inspect --csv-dir /path/to/csv/exports job JOBID report > $SCRATCH/report.json

Useful flags:

Flag	Position	Purpose
--cache-dir DIR	global (before job)	Persist the discovery snapshot and per-module results so repeat calls skip the day-scan and re-computation.
--start ISO --end ISO	job group (after JOBID)	Provide the job window directly to skip job discovery entirely (much fewer queries). Use with --interval when the sampling interval is known.
--interval SECONDS	job group	Override discovered sampling interval.
--refresh	job group	Force fresh discovery, ignoring any cached snapshot.
--cv-threshold 0.05	report subcommand	CV value above which a variance drill-down is reported (default 0.05).
--verbose	report subcommand	Include full per-node / per-GPU arrays under stats.variance.by_node[*].all and stats.variance.by_gpu[*].all, and expand stats.kernels.top from the top 10 to all num_kernels kernels.

That single invocation produces everything the report card needs.

Job-context flexibility

omnistat-inspect resolves the job window in one of two ways:

• Discovery (default) — omnistat-inspect --tsdb-url $URL job JOBID report scans the TSDB for the job's time range. Add --cache-dir $SCRATCH/cache so a second call (e.g. an iterations follow-up) rehydrates from the cached snapshot instead of re-scanning.

• Direct window — when you already know the job's start/end (e.g. from the scheduler), pass them to skip discovery:

  omnistat-inspect --tsdb-url $TSDB_URL job JOBID \
      --start 2026-01-01T12:00:00Z --end 2026-01-01T14:30:00Z --interval 10 \
      report > $SCRATCH/report.json
  

  The query_stats.total_queries field will be noticeably lower than a discovery run. (On the TSDB backend the overview still fills hosts/versions live; on CSV the direct path reports only what the window contains.)

JSON Schema Reference

report.json is an envelope wrapping the report-card payload. The envelope keys are:

• jobid — the analyzed job id
• generated_at — ISO 8601 UTC timestamp of when the report was produced
• data_source — {type: "tsdb"|"csv", url?, dir?}; url is present for TSDB, dir for CSV. This is the source of the "Database" line in Report Metadata.
• overview — job identity/topology (below)
• stats — gauges, counters, hardware counters, and variance (below)
• health — data-collection coverage and health indicators (below)
• query_stats — {total_queries, total_query_time_seconds, elapsed_seconds}; the source of the Report Metadata counts.

Note the nesting: variance lives under stats, and both the data-collection table and the health indicators live under health. There is no top-level variance, data_collection, or metadata key.

overview

• jobid, user, partition
• start_time, end_time (ISO 8601 UTC)
• duration_seconds, duration_human (e.g. "2h 34m 12s")
• num_nodes, num_gpus, hosts[]
• gpu_type, driver_version, vbios_version — each is always a sorted list of distinct strings (single-element in the common case), or null if the field was not present in the data.
• omnistat_version, sampling_interval (seconds)
• annotations[] — strings; empty list if none
• figure_of_merit — list of {name, instance, min, max, last, num_points} or null

stats.gauges[]

Rows in display order. Each entry: {source, label, name, mean, min, max, unit, n, cv, percentiles}. n is the size of the population behind cv / percentiles — pooled sample count for plain gauges, per-node-rate count for counter-derived rate gauges (Network RX/TX rate). min/max are absolute sample extremes over that same population (compare with stats.variance.by_*[].min/max, which are extremes of per-key reduced values — see the reduction field there). percentiles is {p5, p25, p50, p75, p95} over the pooled population. Only metrics that were present in the data are emitted — absent metrics are simply omitted from the list.

Units in the JSON are each metric's native unit (W, MHz, °C, %, bytes, KiB, B/s, KiB/s, J); labels are unit-free. The renderer is responsible for picking a sensible display unit per row (see "Unit selection" below).

stats.counters[]

Each entry: {source, label, name, total, unit}. Same base-unit convention as gauges. Same emit-only-if-present rule.

stats.hardware_counters

null or {rows: [...], flops: [...] | null, variance: {...}}.

Each rows[] entry: {counter, total, active_rate, effective_rate, observed_span_seconds, monotonic, num_series}. Counters are summed per GCD ((instance, card)) over the full job range using reset-aware increase() semantics, so total is robust to the ROCm spurious-zero glitch. active_rate = total ÷ observed_span_seconds (mean per-GCD span actually accumulating); effective_rate = total ÷ job duration (charges startup/activation idle). monotonic is false when a downward step (spurious zero, genuine reset, or multiplexing) was detected. Totals/FLOPS assume a cumulative counter; time-multiplexed counters are rate-like (the series churns instead of growing) and are not yet supported, so an implausible total/FLOPS or a monotonic: false row likely indicates multiplexing rather than a usable figure.

Each flops[] entry: {precision, kind, total_flops, active_rate_flops_per_s, effective_rate_flops_per_s} (or the whole flops is null). active_rate_flops_per_s divides by the mean per-GCD active span, effective_rate_flops_per_s by full wall time.

variance carries the per-counter-name spatial variance of per-GCD counter totals, with the same outer shape as stats.kernels.variance:

{
  "cv_threshold": <float>,
  "metric": "counter_total",     // the compared quantity = per-GCD cumulative counter delta
  "by_node":   [ ... ],          // key fields: {instance}
  "by_gpu_id": [ ... ],          // key fields: {card}
  "by_gpu":    [ ... ]           // key fields: {instance, card}
}

Each list entry has the same outer shape as a stats.variance.by_* entry (source/label/name/unit/reduction/n/cv/min/max/all|percentiles) plus a counter field holding the counter name; source is "GPU", label/name are the counter name, unit is "count", and reduction is "total". The compared quantity is each GCD's cumulative counter total — a direct FLOPS-imbalance proxy, since FLOPS scale with the hardware-counter total, so a single straggler GCD surfaces here. All three lists are [] when nothing crossed cv_threshold.

stats.kernels

null when kernel tracing was off / no kernel data exists (the report omits the Top kernels table and the folded kernel-dispatch-duration variance rows). Otherwise:

{
  "num_kernels": <int>,            // distinct kernels seen across the job
  "total_dispatches": <int>,       // summed over all kernels/GPUs
  "total_duration_ns": <float>,    // summed GPU time over all kernels (ns)
  "dropped_dispatches": <int>,     // node-level dropped count (0 if none)
  "top": [                         // top TOP_KERNELS_LIMIT (10) by default, or all num_kernels kernels under --verbose; ranked by total_duration_ns desc
    { "kernel": "<full mangled name>", "total_duration_ns": <float>,
      "dispatches": <int>, "mean_duration_ns": <float> }
  ],
  "variance": {
    "cv_threshold": <float>,
    "metric": "mean_dispatch_duration_ns",   // the compared quantity = Δduration_ns / Δdispatch_count
    "by_node":   [ ... ],   // key fields: {instance}
    "by_gpu_id": [ ... ],   // key fields: {card}
    "by_gpu":    [ ... ]    // key fields: {instance, card}
  }
}

kernel names are long mangled C++ symbols (600+ bytes) — keep full names in JSON, truncate only at render time. total_duration_ns and mean_duration_ns are native nanoseconds (the renderer converts). A kernel's share of total GPU time is total_duration_ns / stats.kernels.total_duration_ns (compute it at render time — there is no stored percentage field).

The three spatial variance lists (by_node/by_gpu_id/by_gpu) carry exactly the same outer shape as stats.variance.by_* entries (source/label/name/unit/reduction/n/cv/min/max/all|percentiles) plus a kernel field holding the full kernel name; source is "GPU", label is "Mean dispatch duration", unit is "ns", and reduction is "ratio" (Δduration ÷ Δdispatches). The compared quantity is each kernel's mean dispatch duration (ns/dispatch); it is a counter ratio rather than a temporal mean, but the outer entry shape matches a gauge entry, so these fold directly into the matching gauge variance subsection. All three variance lists are [] when nothing crossed cv_threshold. Only the top kernels participate in variance (under --verbose, "top kernels" is the full kernel set, so variance is computed for every kernel).

stats.variance

Nested under stats (not a top-level key).

• cv_threshold, verbose
• by_node, by_gpu_id, by_gpu: flat lists of per-metric entries (empty list when nothing varied along that axis).

Every entry, in every section, carries the same outer shape:

{
  "source": "...", "label": "...", "name": "...", "unit": "...",
  "reduction": "temporal_mean" | "rate" | "ratio" | "total",
  "n": <int>, "cv": <float>,
  "min": {<key_fields>, "value": <float>},
  "max": {<key_fields>, "value": <float>},

  // exactly one of the following, based on n (both in --verbose):
  "all": {<key>: <value>, ...}                // when n <= 16
  "percentiles": {p5, p25, p50, p75, p95}     // when n > 16
}