Run the full DEFT AOI improvement loop for NVIDIA TAO VisualChangeNet / ChangeNet PCB inspection models: baseline evaluate, RCA, Cosmos AnomalyGen / AMP synthetic defects, k-NN mining, retraining, and deployment gating against a customer-defined primary metric and optional constraints. Supports air-gapped/offline runs with pre-staged assets. Use for prompts like "run the DEFT loop", "fine-tune until the configured quality metric meets its target", "optimize a customer metric", or "improve my AOI ChangeNet model with RCA and synthetic defects"; do not use for standalone TAO training, one-off inference, generic anomaly generation, or RCA-only analysis.
Instrucciones de origen · Vista previa de solo lectura
name
tao-run-deft-aoi
description
Run the full DEFT AOI improvement loop for NVIDIA TAO VisualChangeNet / ChangeNet PCB inspection models: baseline evaluate, RCA, Cosmos AnomalyGen / AMP synthetic defects, k-NN mining, retraining, and deployment gating against a customer-defined primary metric and optional constraints. Supports air-gapped/offline runs with pre-staged assets. Use for prompts like "run the DEFT loop", "fine-tune until the configured quality metric meets its target", "optimize a customer metric", or "improve my AOI ChangeNet model with RCA and synthetic defects"; do not use for standalone TAO training, one-off inference, generic anomaly generation, or RCA-only analysis.
Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).
Execution Contract
Treat this as a disk-backed state machine, not as a prose recipe.
Preserve every explicit user value. epoch 1 means num_epochs=1 and
iteration 1 means max_iterations=1; a heuristic or spec default applies
only when the user did not supply that parameter. Show the source of every
run parameter (user, spec, or default) in the Pre-Flight Summary.
Preserve the customer's metric name, operator, target, unit, evaluator, and
constraints. The approved metric_contract is the source of truth for
evaluation, checkpoint selection, completion, and reporting.
After the user approves the Summary, initialize deft_state.json once with
scripts/init_deft_state.py, passing Preflight's exact GPU model/memory,
resolved --network-mode, activation source, and selected absolute Python.
The resulting execution_policy is immutable run state. Never hand-author
or reinitialize it on resume.
Run host Python through scripts/deft_python.sh. On startup, after context
compaction, before every stage, and before any
completion claim, run scripts/deft_context.py --state ... --stage ....
Use its durable next_stage plus the state file's
status, current_iteration, iterations.*.status, stage_completed,
and latest events entry to resume. Do not infer progress from assistant
prose or from an artifact that is not recorded in state.
Invoke the mapped underlying skill after reading the DEFT overlay. Do not
replace a missing/unread stage reference or a failed skill call with guessed
shell commands, inline Python, a different output tree, or data fabricated
from the KPI set.
After initialization, run install/fetch/login/container commands through
scripts/deft_exec.py --state ... -- <command>. Air-gap mode rejects egress
and installs, injects offline flags, and enforces no-pull. Selected
platforms must enforce the equivalent policy.
Commit every stage with scripts/commit_stage.py; it verifies the stage's
required inputs and atomically updates both the resume snapshot and ordered
array inside . Never edit the state file with
inline Python, jq, heredocs, or an editor. Fix rejected evidence; never
fabricate state. For evaluate, pass the metric result,
checkpoint, inference CSV, and threshold directly to .
Pass positive measured from backend elapsed time or a host
timer; missing/zero durations are rejected.
events
deft_state.json
commit_stage.py
--duration-sec
Claim the loop complete only after scripts/finalize_run.py creates the
handoff artifacts, successfully commits loop_stop, and a
fresh read of deft_state.json shows status == "complete",
iterations.baseline.status == "complete", and the final iteration's
status == "complete". A checkpoint, inference CSV, report, or assistant
message is not completion evidence by itself.
Context Discipline
Load references just in time. Re-read state, then read only the current
stage's named section and act.
Never preload/cat every reference or underlying skill, recursively list the
skill tree, or re-read a reference already present in the current context.
Redirect verbose train, inference, Docker, and SDG output to files. Inspect
at most the final 40 lines or a one-line status/artifact check; never print a
full spec, state file, loop log, or generated script into the conversation.
A Skill-tool call loads stage instructions; it does not start a background
orchestrator. Continue the documented stage in the parent immediately after
it returns. Never sleep or poll waiting for a Skill-tool process. For actual
background Docker work, save the PID and poll at intervals no longer than 30s.
At the start of Pre-Flight, resolve network mode before dependencies. Read
exactly one branch: references/air-gap.md for air-gap mode or
references/network-bootstrap.md for network-enabled mode. Never load the
network bootstrap in an air-gapped run.
When to Use This Skill
Use this skill when the user wants an agent to run the full DEFT AOI improvement loop for an NVIDIA TAO VisualChangeNet / ChangeNet PCB inspection model: baseline evaluation, RCA, synthetic defect generation, data mining, retraining, and deployment gating until a KPI target is met.
"Run the DEFT loop"
"Fine-tune until the configured quality metric meets its target"
"Optimize a customer-defined metric while preserving its constraints"
"Improve my AOI ChangeNet model using RCA and synthetic defects"
"Iterate training until the deployment KPI meets the target"
Do not use this skill for a single standalone TAO training run, one-off inference, generic anomaly generation, or RCA-only analysis. Use the relevant agent directly when the user asks for only that step.
Base Model
The loop operates on NVIDIA TAO Visual ChangeNet classify with the NVIDIA C-RADIOv2-B backbone, fine-tuned end-to-end. The architecture is defined in specs/baseline_spec.yaml — that file is the source of truth. Pretrained weights originate from HuggingFace (HF_TOKEN required for networked fetches); NGC_KEY gates container pulls. ChangeNet backbone resolution and the staged-file/HuggingFace-download fallback for model.backbone.pretrained_backbone_path are owned by references/visual-changenet.md; the spec itself must always point to a local mounted file, never a URL. SigLIP for k-NN mining is owned by references/tao-mine-aoi-images.md. AnomalyGen-side checkpoints (Cosmos-Predict2, T5, NVDINOV2, C-RADIO-V3, DINOv2-large, SAM2, Qwen3-VL — ~22 GB for 2B-only, ~140 GB with 14B + T5-11b) live under <workspace>/augmentation/anomalygen/base_checkpoints/. Network-mode rules and staged-asset requirements are in references/air-gap.md; model-specific bootstrap details are in references/paidf-anomalygen.md.
Train AutoML Policy
DEFT AOI owns the iterative data-improvement loop, retraining cadence, and KPI
checkpoint selection. For this workflow only, bypass model-level AutoML even
when the underlying Visual ChangeNet model metadata has automl_enabled: true.
automl_policy: off is a workflow argument to the Visual ChangeNet skill
invocation (the value the parent passes when calling tao-skill-bank:tao-train-visual-changenet
via the Skill tool), not a TAO spec field. Two cases:
Direct docker run visual_changenet train -e <spec> (the path this workflow
actually uses inline): no action needed. The TAO entrypoint is plain training
by default; AutoML lives behind a different code path that the SDK orchestrates.
Effectively, every direct docker run is already automl_policy: off.
SDK-orchestrated dispatch (Brev/SLURM/k8s with the SDK building the
command): pass automl_policy: off to VisualChangeNetSDK.train(...) or the
equivalent runner argument. The SDK uses it to pick the plain-train command
instead of the AutoML wrapper.
Never add automl_policy or a workflow key to the spec YAML. TAO's Hydra
ExperimentConfig schema does not recognize these keys and the train job
fails at config-merge time with
Error merging '<spec>.yaml' with schema: Key 'workflow' not in 'ExperimentConfig'.
This is a workflow-level override only; do not change model metadata, and do
not apply this policy to other workflows.
Launch Intake
After the user confirms they want to run this workflow, ask which supported
platform they intend to run on. Discover the execution platforms from the
installed platform skills (tao-run-on-docker / -slurm / -kubernetes / -brev,
plus any external one); on a runtime that surfaces only the core router skills,
read skills/platform/tao-run-on-*/SKILL.md frontmatter.
After platform selection, read the chosen platform skill's ## Credentials
section and references/skill_info.yaml (required_credentials /
credential_groups).
Never ask for or print credential values. Check only whether the variable is
set ([ -n "$VAR" ] && echo SET || echo UNSET); if unset, name it so the user
can export it or put it in a user-approved env file (~/.tao/secrets.env,
~/.config/tao/.env, or one they point at), loaded with
set -a; source /path/to/.env; set +a. The run never creates or writes that
file.
Agent Behavior
There is exactly one user gate: pre-flight confirmation. Print the Pre-Flight Summary
(see references/preflight.md → Pre-Flight Summary), then STOP and wait for the user to type "go", "yes",
"looks good", or similar explicit approval. Do not launch any side-effecting step
(docker run, training, SDG, mutations under ${RESULTS_DIR}/) before that approval —
reading specs, listing files, docker image inspect, and populating the summary table
are fine. "Autonomous" describes behavior after this gate, not before it. Do not
skip the gate even if the user's original prompt sounded urgent ("just run it", "go
ahead") — the summary itself is the artifact they need to see before approving.
After the gate, the skill is fully autonomous. Run the entire loop without asking
for confirmation. Do not pause between steps. Do not ask "want me to continue?" — just
continue. Only stop if a step fails with an unrecoverable error or a hard-stop gate
fires. Print a one-line status update at each step milestone so the user can follow
progress.
Auto-mode required. The post-gate loop fires constant side-effecting calls
(docker run, ${RESULTS_DIR}/ writes); without auto-accept / bypass-permissions mode it
stalls on the first prompt. Remind the user at the Pre-Flight Summary to enable auto-mode
(shift+tab) before approving.
Blocker recovery. Before the user gate, select a complete installed host
interpreter through deft_python.sh. If none exists, follow only the
already-selected network-mode reference. Air-gap mode hard-stops without a
package-manager command; network-enabled bootstrap is isolated in
references/network-bootstrap.md.
Apply the network-mode branches in references/air-gap.md; record permitted
fetches and directory creation as
post-approval work, or validate staged assets in air-gap mode. After
approval, fix recoverable blockers yourself, then
resume the Pre-Flight step you were on (<blocker> cleared → resuming step N).
Halt only for what you cannot fix (missing workspace/specs/CSVs/credentials,
empty pool, leakage). A fix is not another user gate.
Non-zero command rule. Never repeat an unchanged failed command and never
switch to an undocumented CLI/module path by trial and error. Read the final
error block (not only the container banner), map it to the loaded stage
reference/underlying skill, make one evidence-based correction, and rerun its
documented verification. If the reference does not cover the failure, commit
status=error and halt instead of improvising a reduced workflow.
Revised plan. If any run parameter changes after the original summary was shown (user imposes a time limit, overrides epochs, changes max_iterations, etc.), always re-run Pre-Flight and show an updated summary before proceeding.
Workflow
Execute the loop in this order (full detail in references/pipeline-and-state.md → Pipeline + Stage Execution):
Pre-Flight. Run every check in references/preflight.md. Resolve workspace, specs, CSVs, checkpoints, container images. Hard stop only on missing input you can't resolve yourself (see ## Agent Behavior → Blocker recovery).
Baseline. If deft_state.json already has iterations.baseline.stage_completed == "train" and a best_ckpt_path pointing at an existing file (the upstream automl-deft-pipeline pre-seeds these from its Phase 1 AutoML winner — see its Phase 1 → Phase 2 handoff), skip the train sub-step and resume at inference -> evaluate against the pre-seeded checkpoint. Otherwise run train -> inference -> evaluate by invoking the tao-skill-bank:tao-train-visual-changenet skill. Evaluate with the approved contract and evaluator in references/metric-contract.md. Either way, then rca by invoking tao-skill-bank:tao-analyze-gaps-visual-changenet. Read references/visual-changenet.md, references/metric-contract.md, and references/tao-analyze-gaps-visual-changenet.md first for DEFT-loop-specific args.
Iterate. For each iteration up to max_iterations, execute Pipeline steps 1-7. Between steps re-read deft_state.json and continue from its stage_completed value; do not print the full state.
Stop when the KPI target is met or max_iterations is reached by running
scripts/finalize_run.py with the matching reason. Hard-stop failures are
committed as errors and are never relabeled as successful loop_stop.
Render automatically.scripts/init_deft_state.py writes the initial
results/DEFT_Loop_Report.html; every successful commit_stage.py call
then refreshes it through the deterministic scripts/render_report.py
post-commit hook. The loop_stop commit therefore produces the final
report even when the parent context is saturated. If a hook reports an
error, run scripts/render_report.py --results-dir "${RESULTS_DIR}"
directly after repairing the named presentation input; never hand-author
report HTML.
All pipeline stages run inline in the parent context. Prefer invoking the underlying tao-skill-bank:* skills directly via the Skill tool, layering DEFT-loop conventions on top via the matching references/*.md file. If the mapped Skill tool is unavailable but Docker, the skill source tree, and the stage reference modules are present, use the documented direct-container fallback in references/scripts-and-agents.md; before the first fallback stage, write execution_path=direct-container to the transcript, and for each fallback stage record the mapped underlying skill name plus the exact direct command used. Preserve the same deft_state.json, artifact, and script-backed report contracts. HTML rendering is not delegated.
Using Bundled Scripts
Run bundled scripts through <skill_root>/scripts/deft_python.sh; do not rely
on harness-specific helpers or a shell export surviving the next tool call.
Resolve every path argument to an absolute host path first. Use
deft_context.py before each stage, deft_exec.py for external execution, and
commit_stage.py for all state writes. See
references/scripts-and-agents.md for script invocations, the automatic
report hook, stage mapping, direct-container fallback, and path invariants.
Stage Reference Modules
Each pipeline stage maps to one underlying skill in the bank; the matching references/*.md file layers DEFT-loop conventions (mounts, output dirs, and commit_stage.py arguments) on top of the skill's generic instructions. Read only the current stage's relevant section, then invoke the skill via the Skill tool or the documented direct-container fallback; never preload all stage references. If a reference file is missing, stop and ask the user to reinstall the plugin. The full stage→reference→skill→ownership table lives in references/scripts-and-agents.md → Stage Reference Modules. The stages: train/evaluate (references/visual-changenet.md), anomalygen (references/paidf-anomalygen.md), rca (references/tao-analyze-gaps-visual-changenet.md), routing (references/tao-route-visual-changenet-samples.md), and data_mining (references/tao-mine-aoi-images.md).
Path rule (invariant). Record absolute host artifact paths under
${RESULTS_DIR}. For ChangeNet direct containers, mount
"$WORKSPACE:/data/workspace" and "$RESULTS_DIR:/results"; specs use
/results/baseline/<stage> or /results/iterN/<stage>. Other stages retain
their reference module's required workspace mount. Never remap the run
directory to /results/iterN.
Data, Pre-Flight, Pipeline, and State references
Topic
Reference
Contents
Air-gap activation and offline execution
references/air-gap.md
Global mode triggers, precedence, prohibited network actions, staged-asset requirements, and Pre-Flight evidence
Bring-your-own-data, data contract, output layout, augmentation pool
references/data-layout.md
No public AOI dataset; full <workspace> input tree, ChangeNet four-column required CSV schema, ${RESULTS_DIR}/ output tree, and the two-source mining-pool table
The 10 ordered Pre-Flight checks, required input max_iterations, all defaults, the full Pre-Flight Summary table + populate commands, and the per-iteration runtime estimate
Baseline pre-seed/skip-train logic, the 7 iteration Pipeline steps, the deft_state.json snapshot + event schema, post-stage check, per-iteration HTML render, and the loop-end sequence
Available Scripts table, deterministic report renderer and post-commit hook, Stage Reference Modules table, path-rule invariant, AutoML-policy spec trap
Required input — max_iterations. No default; ask the user if not supplied and do not proceed past Pre-Flight without it. If the user gives a time limit instead, convert it to an estimated max_iterations using the per-iteration runtime figure in references/preflight.md and surface the estimate for confirmation. All other run parameters have defaults — never ask about a parameter with a default. The full defaults list and the Pre-Flight Summary the user approves at the single gate are in references/preflight.md.
Gating
Run the full Pre-Flight (references/preflight.md), print the Pre-Flight Summary, then STOP at the one user gate. After approval, run the baseline (with the pre-seed/skip-train logic) and the 7-step iteration Pipeline, all detailed in references/pipeline-and-state.md.
Hard-stop and never auto-retry on: any stage status=error; train/validation leakage; a missing or zero-row mining pool; a failed CSV existence check; silent-drop; and AMP allocation mismatch. The loop stops when the KPI target is met, max_iterations is reached, or an unrecoverable gate fires. Each terminal path commits loop_stop through commit_stage.py, then follows the loop-end sequence in references/pipeline-and-state.md.
