acr-mirror

name

acr-mirror

description

Generates a bash script that bulk-mirrors NVCF images (and optionally helm charts) from NGC to Azure Container Registry by iterating an `image-manifest/manifest.yaml` SBOM. Resolves the NGC source path **per manifest group** from a built-in mapping table that mirrors `image-manifest/source-registry-config.yaml` (see the per-group source mapping below), overridable per-group via `streaming-env.sh`. Two emit modes — `regsync` (declarative, idempotent, parallel; preferred) or `az acr import` loop (no extra deps). Handles scalar and list tag values, skips `helm-*` keys for the image pass, and applies the `<acr>.azurecr.io/<sub-path>/<image>:<tag>` convention so NVCF helm charts (which concatenate `<registry>/<repository>/<image>`) don't render double-slash paths. Use after the `acr` foundation skill has provisioned the ACR and before `nvcf-self-managed-installation`. Trigger when the user mentions "mirror images", "populate ACR", "regsync", "az acr import", "NGC → ACR", "bulk mirror", "image-manifest sync", or "sync NVCF SBOM to ACR".

Bulk mirror NGC → ACR

What this skill produces

A self-contained bash script that:

1. Locates the NVCF image-manifest SBOM (image-manifest/manifest.yaml at the repo root of a streaming-staging clone) — see Manifest sources for how this works from a streaming-staging clone vs an azure-base-only checkout.
2. Iterates the nvcf-self-managed-stack group, emitting one mirror entry per image:tag pair.
3. Skips chart artifacts (keys prefixed helm-) for the image pass. Mirroring them is covered in Helm chart mirroring.
4. Handles both scalar values (bitnami-cassandra: 5.0.6-nv-1) and list values (nvcf-openbao: [2.2.2-nv-1, 2.5.1-nv-1.1.0] during upgrade transitions).
5. Writes either a regsync.yaml + run command (default), or a loop of az acr import commands.

This skill is the operational counterpart to acr: acr creates the registry; acr-mirror populates it.

When to use it

In the aks-bootstrap flow, this is stage 04b — runs after acr (stage 04) creates the registry and before nvcf-self-managed-installation (post-foundation) installs the helmfile bundle, because that install tries to pull images from <acr>.azurecr.io/<repo>/....

It is not part of the mandatory aks-bootstrap cycle — bootstrap is foundation-only. Mirroring is the first post-foundation step.

Inputs

Sourced from streaming-env.sh and the upstream stages' generated/.env.* files.

Required:

Variable	Description	Example
ACR_NAME	Target ACR (from streaming-env.sh)	acrnvcftestwestus3
NGC_API_KEY_FILE	Path to a mode-600 file containing the raw NGC API key	$HOME/.nvcf-secrets/ngc-api-key

Optional (firm defaults):

Variable	Description	Default
MANIFEST_PATH	Path to manifest.yaml. See Manifest sources.	${REPO_ROOT}/image-manifest/manifest.yaml
MANIFEST_REF	Used only when FETCH_FROM_GITLAB=true — tag or SHA of the manifest commit to fetch	main
FETCH_FROM_GITLAB	If true and MANIFEST_PATH doesn't exist, fetch the manifest from omniverse/ovonsm/image-manifest via the GitLab API. Requires GITLAB_TOKEN.	false
NGC_SOURCE_REGISTRY	Source registry host	nvcr.io
TARGET_REPOSITORY	Sub-path under the ACR. Never empty — see Why the sub-path matters. Defaults to ${CLUSTER_NAME} so multiple deployments sharing one ACR don't collide.	${CLUSTER_NAME}
INCLUDE_GROUPS	Comma-separated manifest groups to mirror (subset filter). Empty = all non-metadata groups.	(all)
NGC_SOURCE_REPO_<GROUP>	Per-group override for the NGC source path. Group hyphens become underscores in the env var name (e.g. NGC_SOURCE_REPO_nvcf_self_managed_stack). See Per-group source mapping for built-in defaults.	(defaults below)
MIRROR_TOOL	regsync (recommended) or az_acr_import	regsync
INCLUDE_HELM_CHARTS	Also mirror helm-* chart artifacts (OCI helm)	false
SKIP_KEYS	Comma-separated manifest keys to skip by name — entries that are not OCI artifacts and 404 if mirrored. The skill can't read component_type at runtime (, which carries it, isn't shipped beside ), so this is a hand-maintained list. Default is upstream (an SBOM umbrella label).

Per-group source mapping

The skill ships with these defaults for every group currently in manifest.yaml:

Manifest group	Default NGC source path (under nvcr.io/)
nvcf-self-managed-stack	0651155215864979/ncp-dev
nvcf-compute-plane-stack	0651155215864979/ncp-dev
container_cache	0651155215864979/ncp-dev
gxcache	0833294136851237/nvcf-ncp-staging
lls	0795883928596950
ucc	nvidia/omniverse
ddcs	nvidia/omniverse
storage-api	nvidia/omniverse

These mirror image-manifest/source-registry-config.yaml (the deployment source of truth) — group keys match manifest.yaml exactly (container_cache, not caches). A group present in the manifest but absent from this map is a hard error, not a silent skip; use INCLUDE_GROUPS to scope a partial run.

To override per-group: set NGC_SOURCE_REPO_<GROUP> in streaming-env.sh, replacing hyphens with underscores. Example:

# In streaming-env.sh
export NGC_SOURCE_REPO_nvcf_self_managed_stack="0833294136851237/some-other-team"
export NGC_SOURCE_REPO_storage_api="nvidia/something-else"

A group that's in the manifest but missing from both the built-in defaults AND any override produces a WARN and is skipped — the script doesn't fail, so unknown future groups don't break existing mirrors. Set the env var to include them.

Multiple NGC source orgs — one key may not cover them all

The groups source from more than one NGC org (see the per-group source table above for the current paths). The mirror authenticates to nvcr.io with a single NGC_API_KEY, and access to private orgs is membership-scoped: if the key's NGC user isn't a member of a private source org, the pull returns 403 Forbidden for that org — which looks like a bad or expired key but is actually an org-access gap, and is easy to lose hours to.

Public orgs don't gate on membership. Some groups source from a public NGC org, where any valid NGC key resolves the images — it's the authenticated token that matters, not org membership. Only the private source orgs require membership, so a key that belongs to the private org you source the control-plane groups from will also pull the public-org groups.

Consequences:

1. Usually one key is enough: a key whose user is a member of the private org(s) the manifest sources from covers everything, public-org groups included. Point NGC_API_KEY_FILE at it.
2. Two private orgs need per-org passes: the multi-key problem only bites when the manifest spans two private orgs that no single key is a member of both (e.g. the key belongs to one private org but a group is sourced from a different private org — the failure in the bug). Then run the mirror once per private org — INCLUDE_GROUPS to that org's groups, NGC_API_KEY_FILE to that org's key:

   # Pass 1 — groups sourced from private org A, using org A's key
   INCLUDE_GROUPS="<groups-for-org-A>" NGC_API_KEY_FILE=~/.nvcf-secrets/ngc-key-A ./<script>.sh
   # Pass 2 — groups sourced from private org B, using org B's key
   INCLUDE_GROUPS="<groups-for-org-B>" NGC_API_KEY_FILE=~/.nvcf-secrets/ngc-key-B ./<script>.sh
   # (Public-org groups can ride along in any pass — any valid key pulls them.)
   

This applies only to the mirror step (which pulls from NGC). Once everything lands in the one ACR, the split disappears for the cluster: AcrPull node identity authenticates all pulls, so no per-org pull secrets are needed in-cluster.

Outputs

• Bash: generated/stageNN-<cluster-name>-acr-mirror.sh (NN = next available stage number in the deployment cycle; commonly 04b or 08).
• regsync config (when MIRROR_TOOL=regsync): generated/regsync.yaml — emitted at script runtime; consumed by regsync once -c generated/regsync.yaml.

The script does NOT write a .env.acr-mirror file because nothing downstream consumes its outputs — successful exit is the signal. It does print a final tally line (mirrored N images, M charts) and (in DRY_RUN=true) the list of pairs without executing.

Manifest sources

Two scenarios, in priority order:

From a streaming-staging clone

image-manifest/manifest.yaml ships at the repo root, pre-synced from omniverse/ovonsm/image-manifest (alongside a COMMIT file for provenance and source-registry-config.yaml). The default MANIFEST_PATH=${REPO_ROOT}/image-manifest/manifest.yaml resolves correctly. To advance to a newer manifest, pin a different streaming-staging tag/SHA and git pull.

From an azure-base-only checkout

No manifest ships with azure-base. Two options:

1. Set MANIFEST_PATH in streaming-env.sh to a local manifest checkout (e.g., a sibling clone of omniverse/ovonsm/image-manifest).
2. Set FETCH_FROM_GITLAB=true and provide GITLAB_TOKEN. The script fetches the manifest from the GitLab API at MANIFEST_REF — equivalent to what streaming-staging's sync-nvcf.sh does, but at deploy time.

The script echoes manifest source: <path> and the version / generated_at fields so the source is auditable from logs.

Why the sub-path matters

Two reasons the sub-path cannot be empty or static:

1. Empty breaks helm pulls. NVCF helm charts concatenate <registry>/<repository>/<image>:<tag>. The helmfile env file's global.image.repository is set to match TARGET_REPOSITORY. If TARGET_REPOSITORY is empty, the chart renders <acr>.azurecr.io//<image>:<tag> (double slash), which docker pull and many container runtimes reject.
2. A static sub-path collides across deployments. When two clusters share one ACR (team registry, prod ACR, dev/staging side-by-side), both mirroring to <acr>/nvcf/<image>:<tag> race to overwrite tags — last write wins, and if the two manifest commits pin different versions of the same image, one cluster pulls the wrong tag.

The default ${CLUSTER_NAME} gives every deployment its own namespace under the ACR. After mirror, set in the helmfile env file:

global:
  image:
    registry: <acr>.azurecr.io                 # not nvcr.io
    repository: aks-nvcf-validation-westus3    # matches TARGET_REPOSITORY (= ${CLUSTER_NAME})
  helm:
    sources:
      registry: <acr>.azurecr.io
      repository: aks-nvcf-validation-westus3

(See nvcf/nvcf-self-managed-stack/environments/base.yaml for the placeholder values that must be replaced.)

No image-pull secret needed once you point at the ACR. The acr skill granted the AKS kubelet identity AcrPull on this registry, so a pull from <acr>.azurecr.io/... is authenticated by the node identity. Once global.image and global.helm.sources point at the ACR you can drop the entire NGC pull-secret apparatus the NGC-direct path requires: the per-namespace nvcr-pull-secret, the global.imagePullSecrets entry, and the docker login/helm registry login nvcr.io steps. This also removes the multi-org pull-secret problem (function pods otherwise pull several nvcr.io orgs, each needing a different key) — everything is one AcrPull-authenticated registry. See the "Pull from a registry mirror (ACR/ECR)" section of the nvcf-self-managed-installation skill for the consumer-side env + step changes.

Override TARGET_REPOSITORY only when the ACR has an existing namespacing convention (e.g., prod/<release-name>) or when multiple deployments must share tags.

Manifest schema (consumed by this skill)

version: '1.0'
generated_at: '2026-05-28T22:34:11Z'
nvcf-self-managed-stack:
  bitnami-cassandra: 5.0.6-nv-1                       # scalar
  nvcf-openbao:                                       # list (upgrade transition; rare)
    - 2.2.2-nv-1
    - 2.5.1-nv-1.1.0
  helm-nvcf-cassandra: 0.11.1                         # chart artifact — skipped in image pass
gxcache:
  gxcache-init: 1e47f722
  gxcache-service: b206ce39
  helm-gxcache: 0.8.2
lls:
  gdn-streaming: 2.0.1
  streaming-proxy: 2.0.1
storage-api:
  helm-discovery-service: 2.3.8
  storage-service: 1.0.2
# ... etc. (ucc, ddcs)

Generation rules:

• Top-level meta keys (version, generated_at) are not groups — skip.
• Every other top-level key is a group. Resolve its NGC source path via the per-group mapping (built-in default or env override).
• Within each group:
  • Keys starting with helm- are OCI helm charts. Skip in the image pass. If INCLUDE_HELM_CHARTS=true, mirror them via OCI helm push (see Helm chart mirroring).
  • All other keys are container images. The key IS the image name.
  • The value is either a string tag (scalar) or a list of tags. Mirror each tag separately.

Bash snippet (reference)

#!/bin/bash
set -euo pipefail

# Resolve repo root from script location — works regardless of CWD.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
source "$REPO_ROOT/streaming-env.sh"
if [[ ! -f "$REPO_ROOT/generated/.env.acr" ]]; then
  echo "ERROR: $REPO_ROOT/generated/.env.acr not found. Run the acr skill (stage 04) first." >&2
  exit 1
fi
source "$REPO_ROOT/generated/.env.acr"

: "${ACR_NAME:?}" "${NGC_API_KEY_FILE:?}"
[[ -f "$NGC_API_KEY_FILE" ]] || { echo "ERROR: NGC_API_KEY_FILE='$NGC_API_KEY_FILE' is not a file" >&2; exit 1; }

NGC_SOURCE_REGISTRY="${NGC_SOURCE_REGISTRY:-nvcr.io}"
TARGET_REPOSITORY="${TARGET_REPOSITORY:-$CLUSTER_NAME}"
[[ -n "$TARGET_REPOSITORY" ]] || { echo "ERROR: TARGET_REPOSITORY is empty (no CLUSTER_NAME fallback). Set it in streaming-env.sh." >&2; exit 1; }
MIRROR_TOOL=""
INCLUDE_HELM_CHARTS=
DRY_RUN=






SKIP_KEYS=
IFS=  -ra SKIP_KEYS_ARR <<< 
MANIFEST_PATH=




FETCHED_MANIFEST=
CHARTS_DIR=
() {
  [[ -n  && -f  ]] &&  -f 
  [[ -n  && -d  ]] &&  -rf 
  
  
  
   0
}
 cleanup EXIT








 -A DEFAULT_SOURCE_REPO=(
  []=
  []=
  []=
  []=
  []=
  []=
  []=
  []=
)
() {
   group= var=
    
}


 [[ ! -f  ]]; 
   [[  ==  ]]; 
    : 
    MANIFEST_REF=
     
    FETCHED_MANIFEST=$( -t acr-mirror-manifest.XXXXXX)
    curl -fsSL -H  \
       \
      -o 
    MANIFEST_PATH=
  
      >&2
      >&2
      >&2
      >&2
     1
  


 -v yq >/dev/null  || {   >&2;  1; }



yq --version 2>&1 | grep -qi mikefarah || {
    >&2
    >&2
   1
}
 
yq -r  


 -t ALL_GROUPS < <(yq -r  )





 [[ -n  ]]; 
  IFS=  -ra REQUESTED <<< 
  SELECTED_GROUPS=()
   g  ; 
     r  ;  [[  ==  ]] && SELECTED_GROUPS+=() && ; 
  

  SELECTED_GROUPS=()

 


 
az acr login --name 
 [[  ==  ]]; 
   
  helm registry login  -u  --password-stdin < 




TUPLES=()  
 group  ; 
  source_repo=$(resolve_source_repo )
   [[ -z  ]]; 
    
    
    
      >&2
     1
  
   IFS=  -r pair; 
    image=; tag=
    skip=;  s  ;  [[  ==  ]] && skip=1 && ; 
     [[ -n  ]]; 
        >&2
      
    
    TUPLES+=()
   < <(yq -r  )

 



 [[  -eq 0 && -z  ]]; 
    >&2
    >&2
   1


 [[  ==  ]]; 
   -p 
  CONFIG=
  {
     <<
     tuple  ; 
      IFS=$  -r group image tag source_repo <<< 
       <<
    
  } > 
   

   -v regsync >/dev/null || {  ;  1; }
   NGC_API_KEY
  NGC_API_KEY=; NGC_API_KEY=
   [[  ==  ]];  regsync check -c ;  regsync once -c ; 
   NGC_API_KEY

 [[  ==  ]]; 
  
   tuple  ; 
    IFS=$  -r group image tag source_repo <<< 
    src=
    dst=
     [[  ==  ]]; 
       
    
       
      az acr import --name  \
        --  --image  \
        --username  --password  \
        --only-show-errors
    
  

    >&2
   1



 [[  ==  ]]; 
   
  CHARTS_DIR=$( -d -t acr-mirror-charts.XXXXXX)
   group  ; 
    source_repo=$(resolve_source_repo ); [[ -z  ]] && 
     IFS=  -r pair; 
      chart=; tag=
       [[  ==  ]]; 
         
      
        helm pull  --version  -d 
        helm push  
         -f 
      
     < <(yq -r  )
  


 

Helm chart mirroring (optional)

Charts in the manifest are keys prefixed helm- (e.g., helm-nvcf-cassandra: 0.11.1). They are OCI helm artifacts at the source — they need helm pull + helm push, not regsync or az acr import. The script above handles this when INCLUDE_HELM_CHARTS=true.

Why optional: if the cluster has outbound access to nvcr.io and a working NGC auth secret, the helmfile install pulls charts directly from NGC at install time — no mirror needed. Mirror them only for full air-gap where the cluster must not reach nvcr.io.

When mirroring charts, update the helmfile env file:

global:
  helm:
    sources:
      registry: <acr>.azurecr.io
      repository: <target-repository>   # same sub-path as the images

Verification

After mirror, spot-check:

# Image:tag count vs SBOM (across all groups; same exclusion as the mirror — skip helm-*).
# Both counts are at the tag level — an image with 2 tags (e.g. nvcf-openbao
# during an upgrade transition) contributes 2 to each side.
SBOM_COUNT=$(yq -r '
  to_entries | .[]
  | select(.key != "version" and .key != "generated_at")
  | .value | to_entries | .[]
  | select(.key | test("^helm-") | not)
  | (.value | [.] | flatten | .[])
  | "x"
' "$MANIFEST_PATH" | wc -l | tr -d ' ')

ACR_COUNT=0
while IFS= read -r repo; do
  [[ -z "$repo" ]] && continue
  tag_count=$(az acr repository show-tags --name "$ACR_NAME" --repository "$repo" -o tsv 2>/dev/null | wc -l | tr -d ' ')
  ACR_COUNT=$((ACR_COUNT + tag_count))
done < <(az acr repository list --name "$ACR_NAME" -o tsv | grep "^${TARGET_REPOSITORY}/")

echo "SBOM expects: $SBOM_COUNT image:tag pair(s)  /  ACR has: $ACR_COUNT under ${TARGET_REPOSITORY}/"

# Spot-check a tag from any group
az acr repository show-tags --name "$ACR_NAME" --repository "${TARGET_REPOSITORY}/bitnami-cassandra" -o tsv

If counts diverge, re-run with DRY_RUN=true to inspect the planned sync list (each entry is prefixed with its [group]), then re-run for real.

Teardown

Mirror state is just data in ACR. Either:

• az acr repository delete --name "$ACR_NAME" --repository "<sub-path>/<image>" --yes per image (when retiring specific images), or
• Let the ACR itself be torn down by the acr skill's --teardown (which destroys the registry and everything in it).

There is no acr-mirror --teardown because the deliberate destructive action lives in the foundation skill.

Validation checklist

• Script opens with set -euo pipefail and resolves repo root via BASH_SOURCE
• Sources streaming-env.sh and .env.acr (guard: error if stage 04 hasn't run)
• Required vars (ACR_NAME, NGC_API_KEY_FILE) have fail-fast checks; no other required inputs
• NGC_API_KEY_FILE is verified to be a file ([[ -f ]]) before any redirection — the leak-postmortem pattern from earlier in this project
• DEFAULT_SOURCE_REPO table is present and covers every group currently in manifest.yaml (nvcf-self-managed-stack, nvcf-compute-plane-stack, container_cache, gxcache, lls, ucc, ddcs, storage-api) and matches source-registry-config.yaml
• An unmapped non-metadata group is a hard error (exit 1), not a WARN-skip
• cleanup() ends with return 0 so the EXIT trap can't fail a successful run
• resolve_source_repo honors per-group env override (NGC_SOURCE_REPO_<GROUP> with hyphens→underscores) before falling back to the default table
• Unknown groups emit WARN and are skipped — the script does NOT fail on them
• Manifest path resolves with the streaming-staging-clone default; fallback to GitLab fetch is gated behind FETCH_FROM_GITLAB=true
• Tempfiles for the fetched manifest and helm-charts directory are allocated via mktemp / mktemp -d (never fixed /tmp/<name> paths) and a single trap cleanup EXIT removes them on every exit path — safe under concurrent runs
• Manifest provenance is echoed (version, generated_at) for audit
• Top-level metadata keys (, ) are NOT treated as groups

References

Topic	Source
omniverse/ovonsm/image-manifest (SBOM authority)	https://gitlab-master.nvidia.com/omniverse/ovonsm/image-manifest
Helmfile env placeholder values (where NGC paths are replaced with ACR)	streaming-staging/nvcf/nvcf-self-managed-stack/environments/base.yaml
Streaming-staging sync script (mirrors the manifest into the bundle)	streaming-staging/sync-nvcf.sh
regsync (regclient) docs	https://regclient.org/usage/regsync/
az acr import	https://learn.microsoft.com/cli/azure/acr#az-acr-import
Why per-repo auth (repoAuth: true) for NGC	NGC namespace returns fresh challenges per repo; cached token mishandles this