Full PAIDF AnomalyGen pipeline — fine-tune on a new anomaly dataset, generate synthetic anomaly images (SDG), evaluate quality (nn_score), and search per-sample (guidance, crop_ratio) parameters. Three modes: full (Phase 0→7: finetune then generate), finetune_only (Phase 0→1: train only), inference_only (Phase 0, 2→7: generate from an existing checkpoint). Use when the user asks to "fine-tune AnomalyGen", "generate anomaly images", "run PAIDF SDG", "evaluate SDG output quality", "run per-sample search", or run any part of the AnomalyGen pipeline, even if they only mention one phase.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Requires docker + nvidia-container-toolkit and a CUDA GPU. Pulls the `metropolis_sdg.paidf_anomalygen` image declared in `versions.yaml` at the skill bank root.
metadata
{"author":"NVIDIA Corporation","version":"0.1.0"}
allowed-tools
Read Bash
description
Full PAIDF AnomalyGen pipeline — fine-tune on a new anomaly dataset, generate synthetic anomaly images (SDG), evaluate quality (nn_score), and search per-sample (guidance, crop_ratio) parameters. Three modes: full (Phase 0→7: finetune then generate), finetune_only (Phase 0→1: train only), inference_only (Phase 0, 2→7: generate from an existing checkpoint). Use when the user asks to "fine-tune AnomalyGen", "generate anomaly images", "run PAIDF SDG", "evaluate SDG output quality", "run per-sample search", or run any part of the AnomalyGen pipeline, even if they only mention one phase.
tags
["tao","data"]
PAIDF AnomalyGen
Multi-phase pipeline (0–7); the mode flag selects which phases run.
Assemble best-of-rounds into searched/ (stitch only), plus rounds/search_summary.csv
full, inference_only
7
Filter searched/ by nn_threshold (default 0.4), regen dropped samples, then canonical bucket eval → searched/{per_sample.csv, eval.log}
full, inference_only
Run every phase through to completion without mid-run pauses. Collect all
required parameters up front, and run every command from the repo root.
Shell setup. All ${ANOMALYGEN_SCRIPTS} references resolve to the packaged
helper-script directory. Inside the container this is preset (ENV ANOMALYGEN_SCRIPTS=<dir>/scripts/utilities); on the host, export it once per
shell:
python3 -m scripts.utilities.<name> invocations work from any CWD inside the
container (PYTHONPATH is preset) and from the repo root on the host. When inside
a product container (ANOMALYGEN_PRODUCT_MODE=1), invoke anomalygen-guard
before any GPU work; if it reports BLOCKED, fix the listed issues before
continuing.
Quick Start
The pipeline runs inside the metropolis_sdg.paidf_anomalygen container
(declared in versions.yaml) or any host with the cosmos-predict2 conda env
active. All phase commands assume that environment, at the repo root, with
ANOMALYGEN_SCRIPTS exported.
Minimal end-to-end run (mode=full):
# 1. Set the shared variables (see "Shared variables" for the full set).export ANOMALYGEN_SCRIPTS="$(git rev-parse --show-toplevel)/scripts/utilities"
MODE=full
NAME=my_exp
DATASET_DIR=/data/uc1
DEFECT_DESC=assets/defect_spec_template.jsonl
NUM_SDG=20
MODEL_SIZE=2b
# 2. Phase 0 — verify / download checkpoints (~140 GB; needs HF_TOKEN).${ANOMALYGEN_SCRIPTS}/check.sh || ${ANOMALYGEN_SCRIPTS}/download_checkpoints.sh
# 3. Walk Phases 1→7 in order (see each Phase section).
For mode=inference_only (reuse a checkpoint) also set CKPT/STEP and skip
Phase 1. For mode=finetune_only run only Phases 0–1.
Running in Docker — container launch, mounts & permissions
The paidf-anomalygen image runs as a non-root baked-in user (USER anomalygen, uid=10000), independent of your host uid. Docker does not remap
uids on bind mounts, so a host directory owned by your uid is not writable by
uid 10000 and the container fails the instant it tries to create a file there.
Run as your host uid with --user "$(id -u):$(id -g)" plus the mandatory
/etc/passwd+/etc/group and HOME/cache-redirect companions, and run the
fail-fast write preflight before Phase 0. See references/docker.md for the
full docker run command, the load-bearing-flag table, the preflight snippet,
and the uid-10000 chown/chmod fallback.
Reference files — read before executing phases
Read references/finetune.md before Phase 0/1 and references/inference.md
before any of Phases 2–7; for mode=full read both before starting. The
remaining references below are on-demand — read when troubleshooting or needing
full detail for a specific phase.
File
Read when
references/finetune.md
Before Phase 0/1: env check, checkpoint download, dataset validation, config generation, training commands, best-checkpoint selection
references/finetune-commands.md
Exact Phase 1 Step 1–4 commands and CKPT/STEP derivation
references/inference-commands.md
Exact Phase 5 run_round.sh and Phase 7 filter_with_regen commands
num_SDG allocation depends on prep_testcase.sh --mode: inference (default,
Phase 2) is uniform across defect types, override per-defect via
--per-defect-counts; validation (Phase 1's validation JSONL) is proportional
to training mask counts (largest-remainder rounding) and enforces ≥1 per defect.
See references/prep-testcase.md for the full mode table.
Parameter
Description
mode
full (Phase 0→7), inference_only (skip Phase 1), or finetune_only (Phase 0→1 only).
name
Experiment label.
dataset_dir
Training/reference dataset root. Drives mask-count allocation, AMP submask templates, and holds semantic_segmentation_labels.json for cad defects.
defect_spec
JSONL tagging each defect spatial_dependency as free/text/cad. text entries need roi_prompt_defect_location. Template: assets/defect_spec_template.jsonl.
num_SDG
Total output samples per bucket. (Ignored when mode=finetune_only.)
Conditionally required
Parameter
Required when
Description
checkpoint_dir / step
mode=inference_only
Pre-existing fine-tuned model. In mode=full these are auto-derived after Phase 1; passing them is an error. In mode=finetune_only silently ignored — Phase 1 always trains from scratch (no resume-from-checkpoint support). Both must be present together — supplying only one is an error.
Optional parameters
Parameter
Default
Description
clean_dir
dataset_dir
Clean images. Set only when they live outside the training dataset. Forwarded as --clean-dir to prep-testcase and --clean-image-path to finetune.
validation_jsonl
auto-generated
Pre-built validation JSONL for Phase 1. When supplied, preflight verifies every defect_spec type appears and paths exist.
num_search_run
3
Per-sample search budget for Phase 5. 0 skips search (only original/). (Ignored when mode=finetune_only.)
nn_threshold
0.4
nn_score cutoff for Phase 7 (DINOv2 correspondence to real defects — key KPI). Samples below are regenerated; final searched/ always has num_SDG. 0 disables filtering.
mode=inference_only missing either checkpoint_dir or step → halt: "inference_only requires both checkpoint_dir and step."
mode=full with checkpoint_dir or step supplied → halt: "full mode runs finetune; use mode=inference_only to reuse an existing checkpoint."
Shared variables
Set once before Phase 0:
MODE=<full|inference_only|finetune_only>
NAME=<exp>
DATASET_DIR=<dataset_dir>
CLEAN_DIR=${clean_dir:-${DATASET_DIR}}
CKPT=<checkpoint_dir> # required iff MODE=inference_only; auto-derived after Phase 1 when MODE=full
STEP=<iter> # required iff MODE=inference_only; auto-derived after Phase 1 when MODE=full
NUM_SDG=<N>
DEFECT_DESC=<defect_spec.jsonl>
DEFECTS=(T+A T+B) # TEXTURE+TYPE names. For mode=inference_only, derive from ${CKPT}/ag_config.yaml → dataloader_train.dataset.anomaly_types (also printed by validate_checkpoint.py in Phase 0). For mode=full, take from DEFECT_DESC entries. See references/inference.md §Phase 0.
NUM_SEARCH_RUN=${num_search_run:-3}
NN_THRESHOLD=${nn_threshold:-0.4}
MODEL_SIZE=<2b|14b>
NUM_GPUS=${num_gpus:-1}
MAX_ITER=${max_iter:-75000}
SAVE_ITER=${save_iter:-5000}
VALIDATION_ITER=${validation_iter:-5000}
LR=${lr:-0.02}
BATCH_SIZE=${batch_size:-2}
IMAGE_SIZE=${image_size:-512}
VALIDATION_JSONL=${validation_jsonl:-}# optional; set by Phase 1 Step 2 if not user-supplied
BASE=results/${NAME}
JSONL=ag_inference/${NAME}/testcase.jsonl
ORIGINAL=${BASE}/original
SEARCHED=${BASE}/searched
ROUNDS=${BASE}/rounds
REGENS=${BASE}/regens
Guard preflight (product mode only)
When ANOMALYGEN_PRODUCT_MODE=1, run
.agents/skills/anomalygen-guard/scripts/preflight.py before any GPU work and
fix any BLOCKED issues. --validation-jsonl is forwarded only when the user
supplied one; for MODE=finetune_only omit --num-sdg if not supplied. See
references/guard-and-custom-counts.md for the full preflight command with all
forwarded flags and the validation-JSONL / allocate_samples.py 0-entry
checks.
Phase 0 — checkpoints
Read references/finetune.md §Phase 0 for HF_TOKEN requirements and what gets
downloaded (~140 GB). Verify first; download only what is missing.
Phase 1 — fine-tune (skip when MODE=inference_only)
Read references/finetune.md §Phase 1 for dataset structure, config template
details, and best-checkpoint selection. Four steps: (1) validate dataset /
derive anomaly types, (2) generate the validation JSONL (skip if user supplied
VALIDATION_JSONL), (3) generate the training config — show it to the user
and confirm before writing — (4) launch training in the background. Then
derive CKPT (path encodes upper-case MODEL_SIZE) and STEP (highest
nn_score step from validation logs). If MODE=finetune_only, stop after
training. See references/finetune-commands.md for the exact Step 1–4 commands
and the CKPT/STEP derivation snippet.
Phase 2 — prep-testcase (skip when MODE=finetune_only)
Read references/inference.md §Phase 2 for AMP routing detail and n_seeds
sizing. Do NOT pass --seeds — it is auto-computed and is not a recognized
flag. prep_testcase.sh defaults to --mode inference (uniform allocation
across defect types, no KPI floor), which Phase 2 always uses.
Custom per-defect counts: when the user specifies counts per defect type,
translate to --num-sdg plus a --per-defect-counts JSON dict (types absent
from the dict get 0; sum should equal --num-sdg, else the script warns on
stderr and uses the override sum). Confirm the allocation when intent is
ambiguous. See references/guard-and-custom-counts.md for the full
--per-defect-counts command example and the ambiguity-handling detail.
Phase 3 — SDG → original/
Read references/inference.md §Phase 3 for JSONL validation against the
checkpoint, multi-GPU caveats, and output verification.
Read references/inference.md §Eval for score interpretation and feature-count
explanation. run_eval.sh writes per_sample.csv and eval.log inside
original/ and merges nn_score into SDG_result.csv.
Read references/inference.md §Phase 5 for draw strategy, ranges, and re-AMP
guidance. For r in 1..NUM_SEARCH_RUN:
Read prior round's per_sample.csv (or ${ORIGINAL}/per_sample.csv for r=1).
Write ${ROUNDS}/round_${r}/draws.json with selected (guidance, crop_ratio) per sample.
Run round via ${ANOMALYGEN_SCRIPTS}/run_round.sh (SDG + eval; the round dir
gets its own sdg/{SDG_result.csv, per_sample.csv, eval.log}). See
references/inference-commands.md §Phase 5 for the full command and flags.
NUM_SEARCH_RUN=0 is valid — skip this phase entirely and let Phase 6
clone original/ into searched/.
Phase 6 — assemble searched/ (stitch only)
Always run assemble (works with 0 rounds — searched/ clones original/, so
downstream always reads searched/ regardless of num_search_run). Stitch-only:
copies winning images per sample-index into searched/ and carries over
per-sample nn_score / mnn_score from each pick's source-round per_sample.csv.
No eval — Phase 7 emits the canonical searched/eval.log.
Phase 7 runs by default (nn_threshold=0.4) on every mode=full and
mode=inference_only invocation; pass nn_threshold=0 to skip it. It filters
searched/ by nn_threshold, regenerates dropped samples via re-AMP (fresh
(clean, submask) pairing in the same defect type) for up to 5 attempts, then
falls back to best-scoring non-passing regens and finally to dropped originals,
so the final bucket always equals num_SDG.
Run python3 -m scripts.utilities.filter_with_regen. It runs the final
run_eval.sh internally — the only eval against searched/. Read
references/inference.md §Phase 7 for regen mechanics, source-column tracing,
and the regens/regen_summary.csv schema; see
references/inference-commands.md §Phase 7 for the full command and flags.
Output layout
Every bucket that gets eval'd carries the same triad of files:
SDG_result.csv (generation params + nn_score), per_sample.csv
(per-sample nn + mnn), and eval.log (aggregate FID / per-defect avg).
Buckets live under results/<name>/ as original/ (Phase 3+4), searched/
(Phase 6 stitch + Phase 7 filter+regen+eval), rounds/round_NN/ (Phase 5,
plus search_summary.csv), and regens/regen_NN/ (Phase 7, plus
regen_summary.csv).
See references/output-layout.md for the full directory tree with per-file
annotations and the post-run Verification checklist (image counts per
bucket, search_summary.csv / regen_summary.csv row checks, and the per-type
nn_score / mnn_score / fid fields in each eval.log).
Error handling
Common pipeline failure modes (missing mask dirs, short/empty AMP output and
the 0 entries written halt, mid-round SDG failure resume, off-boundary
step) are covered in references/error-handling.md; see also
references/finetune.md and references/inference.md for phase-specific
error handling.
