| name | aicr-uat-report |
| description | Use when reporting on UAT health across services and GPU targets — which
service (EKS/GKE/AKS) x GPU (H100/GB200) x intent combinations are passing
or failing in the UAT Run workflow (uat-run.yaml). Triggers on "UAT report",
"/aicr-uat-report", "which UAT combos are failing", "UAT pass rate",
"download the UAT debug bundle", "why did the UAT run fail", or
RC/release-candidate validation prep that needs the combinations to test
manually. Runs the bundled uat_report.py, classifies failures as product
vs infra signal, prints a summary table plus an RC validation priority
list, and can download the per-run cluster debug bundles for triage.
|
AICR UAT Report
Reports how the UAT Run workflow
(https://github.com/NVIDIA/aicr/actions/workflows/uat-run.yaml) performed
over a lookback window, aggregated by service x GPU x intent, with each
failure classified as product signal (real test failure) or infra noise.
The output feeds the release process: combinations failing on main are
the ones to test more closely during RC validation.
When to Use
- User asks for a UAT report, UAT pass rates, or failing UAT combinations
- User invokes
/aicr-uat-report (optionally with a number of days)
- Release prep needs the list of service/GPU combos to validate manually
- User asks why a UAT run failed, or for the debug bundle behind a failure
Do NOT use this skill to re-run, dispatch, or cancel UAT runs.
Inputs
- days (optional, default 3): lookback window. If the user says
"over the last week", pass
--days 7. Do not ask — default to 3 when
unspecified.
- Only runs against
main (empty aicr_version input) are reported by
default; that is the release-process signal. Add --all-versions only
if the user explicitly asks to compare against release-tag runs.
- debug bundles (optional): pass
--download-debug <dir> when the user
asks why something failed, or when Step 2 classifies a failure as product
signal. Skip it for a plain pass-rate report — bundles are tens of MB.
How the Data Works
The workflow's run-name encodes everything needed — no per-job digging:
UAT <reservation> <intent> @ <version|main>[ #dispatch_key].
Reservation names are <cloud>-<gpu> rows from
infra/uat/reservations.yaml (e.g. aws-h100, gcp-h100, azure-h100,
aws-gb200), and cloud maps to service: aws=EKS, gcp=GKE, azure=AKS,
kind=Kind (self-hosted nvkind lane). Runs before 2026-07-21 used a title
without the intent word; the script derives intent from the nightly
dispatch-key cell index (version-outer/intent-inner, training first) and
marks those rows "(intent derived)".
Debug bundles
Each per-cloud workflow (uat-aws.yaml, uat-gcp.yaml, uat-azure.yaml,
uat-kind.yaml) runs tests/uat/<cloud>/run debug on failure — before
teardown, while the cluster is still up — and uploads
uat-<cloud>[-<intent>]-debug-<run_id> with 30-day retention. Reusable
workflows inherit the caller's run_id, so the artifact hangs off the
uat-run.yaml run the report already lists.
The upload is gated on failure() && steps.prep.outcome != 'skipped', so
no bundle exists when the cloud job died in bring-up or image build, or
when the failure was in a downstream job (evidence ingest) while the cluster
job passed. Absence is itself a classification signal, not an error.
What to open, in triage order (if-no-files-found: ignore, so any entry can
be missing; cluster-debug/ prefix omitted below):
| Open | When / what it answers |
|---|
MANIFEST.yaml | Always first — runId, config, resolved recipe + criteria, and failingChecks lifted from report.json |
report.json | Full validator results; absent if the run died before validate |
train-logs/**, serve-logs/** | A CUJ check failed (NCCL, inference-perf) |
cr-skyhooks.yaml, node-reboot-fingerprint.txt, readiness-gate.log | Readiness-gate or tuning-race failure — Skyhook status.status, taints, bootID/kernel |
pods-notready.txt, events.txt | Scheduling, eviction, OOM |
logs-<namespace>.txt | The operator owning the failing resource |
nodes*, other cr-*.yaml, ns-*.txt | Broader node and operator state |
snapshot.yaml, recipe.yaml, dry-run.json | What was collected / resolved / deployed |
evidence-result.json, evidence/pointer.yaml | Signed-evidence emit outcome |
Procedure
Step 1 — Run the report script
python3 .agents/skills/aicr-uat-report/uat_report.py --days 3
It prints, per version, a Markdown table (Service | GPU | Intent | Pass | Failures) and a "Failure detail" section listing each failing run's
timestamp, URL, and the failed job/step names. It is read-only
(gh run list / gh run view). If gh is not authenticated, stop and
tell the user to run gh auth status.
Step 2 — Classify each failure
Map the failed step name to a failure nature. This drives the RC
priority ranking, so classify every failure:
| Failed step contains | Nature | Product signal? |
|---|
UAT - validate, UAT - prep, CUJ/test phase names | Real test failure | YES — but the validate step also emits signed evidence, so confirm against report.json (Step 2b) before ranking |
Bringup Infra, provision/actuator steps | Infra bring-up failure | no |
Buildx, Build and push, image/GHCR steps | CI/image flake | no |
Validate inputs, helmfile apply, install steps | Setup/install issue | maybe — recurring = investigate |
A retry that went green the same night (same combo, later timestamp,
success) downgrades the earlier failure to a flake.
Step 2b — Pull debug bundles (only for product-signal failures)
Skip this step entirely for a routine pass-rate report. Run it when the
user asks why something failed, or when Step 2 found a test-phase
failure worth root-causing:
python3 .agents/skills/aicr-uat-report/uat_report.py --days 3 \
--download-debug /tmp/uat-debug --max-downloads 3
Prefer --run <id> (repeatable) over raising --max-downloads: usually only
the latest failure per failing combo is worth reading.
Bundles land in <dir>/<service>-<gpu>-<intent>-<run_id>/, each with a
printed digest — MANIFEST head, failing checks from report.json, and a
one-line contents summary. Read that summary for presence, not file names:
a missing evidence/ or report.json says the run died before that stage,
which is often the whole diagnosis. Then open files per the Debug bundles
table above.
Cite file_path:line and the run ID for every finding. Leave the download
directory in place — the user may want to keep digging.
Step 3 — Render the report
Produce exactly two artifacts, in this order (see Output Format
Reference). Sort the table worst-first: lowest pass ratio at the top;
bold the Service/GPU/Intent cells of rows with product-signal failures.
Include run URLs as links for at least the most recent failure of each
failing combo.
Step 4 — Write the RC validation input
A numbered priority list derived from the table:
- Combos with consistent test-phase failures (0/N or repeated
validate-phase failures) — top manual-validation priority.
- Combos whose most recent failure is test-phase (even if earlier
ones were infra) — deserve a close look.
- Combos with infra/CI-only failures — noisy, not product signal;
note them but rank low.
- Reservations in bring-up (e.g. GB200 while
nightly-intents: [],
kind lane) — expected churn, call out separately.
- Green combos — state them explicitly as lowest priority; a clean
bill is information too.
Output Format Reference
## UAT report against `main` (<start>–<end>, N runs)
| Service | GPU | Intent | Pass | Failure nature |
|---|---|---|---|---|
| **AKS** | **H100** | **training** | **0/4** | Real test failures — every run fails at "UAT - validate (all phases)" ([latest](<url>)) |
| EKS | H100 | training | 2/5 | Infra only — 2x bring-up, 1x Buildx CI flake; no test-phase failures |
| GKE | H100 | training | 4/4 | Green |
## RC validation input
1. **<Service>/<GPU>/<intent> is the clear red flag** — <pass ratio,
failure signature, latest run link, whether it also fails on release
tags (env issue) or only main (regression candidate)>.
2. ...
N. **<green combos> are solidly green** — lowest manual-testing priority.
Keep failure-nature cells to one sentence; detail beyond that belongs in
the RC list, not the table.
Failure Modes
gh run list returns nothing — window may predate retention or the
workflow was renamed; say so rather than reporting "all green".- Unparsed titles in the Notes line — the
run-name format in
uat-run.yaml changed; read the workflow's current format string and
update NEW_TITLE/OLD_TITLE in uat_report.py in the same PR.
- Unknown reservation (e.g. new cloud) — the script falls back to the
uppercased cloud token as the service name; cross-check new rows against
infra/uat/reservations.yaml.
- A combo has very few runs (e.g. 0/1) — flag low sample size instead
of declaring it broken.
- "no debug artifact" — expected for bring-up/Buildx/ingest failures
(see Debug bundles). Report it as corroborating the infra classification;
do not present it as a tooling problem.
- "debug artifact expired" — the window exceeds the 30-day retention.
Nothing to recover; note it and work from failed step names.
cluster-debug/ missing or thin — the collector is best-effort and
its cloud credentials can expire on a long failure. Say the bundle is
incomplete; do not read it as the cluster being healthy.- Failing step name disagrees with
report.json — trust the bundle.
UAT - validate (all phases) + emit signed evidence covers two concerns,
so N/N passing checks under a failed step means the evidence leg failed,
not the product. Reclassify to infra before ranking it in Step 4.
What This Skill Does NOT Do
- Does not dispatch, re-run, or cancel workflow runs
- Does not download raw job logs (
gh run view --log); it reports failed
step names and, on request, the uploaded cluster debug bundles
- Does not modify reservations, workflows, or any in-repo file — the only
writes are downloaded artifacts under the
--download-debug directory
