ecr-mirror

name	ecr-mirror
description	Mirror every KAS/NVCF manifest image and optional Helm OCI chart from NGC into AWS ECR using generated regsync config.
version	1.1.0
author	NVIDIA Omniverse Streaming
tags	["aws","ecr","regsync","nvcf"]
tools	["Shell","Read","Write"]

ECR Mirror

Purpose

Populate AWS ECR from image-manifest/manifest.yaml before installing the NVCF self-managed stack on EKS. The manifest is the source of truth: mirror every non-helm image:tag entry, and mirror helm-* OCI chart artifacts when INCLUDE_HELM_CHARTS=true. For AWS this defaults to true.

Use this after AWS infra has created or authorized ECR access. Do not use this skill for Terraform, EKS cluster creation, NVCF installation, function deploys, or stream validation.

Inputs

Required:

Variable	Description
AWS_ACCOUNT_ID	AWS account that owns the target ECR registry.
AWS_REGION	Target ECR region, for example us-west-2.
CLUSTER_NAME	Default ECR repository prefix when TARGET_REPOSITORY is unset.

Optional:

Variable	Default	Description
NGC_API_KEY	unset	Required only when DRY_RUN=false; export in the shell for regsync and never print it.
TARGET_REPOSITORY	${CLUSTER_NAME}	Non-empty ECR repository prefix.
MANIFEST_PATH	${REPO_ROOT}/image-manifest/manifest.yaml	Manifest source of truth.
SOURCE_REGISTRY_CONFIG	beside MANIFEST_PATH	Source registry resolver.
INCLUDE_HELM_CHARTS	true	Include helm-* OCI chart artifacts in the regsync config.
INCLUDE_GROUPS	empty	Comma-separated manifest groups to mirror; empty means all groups.
SKIP_KEYS	nvcf-stack	Comma-separated manifest keys to skip by name — entries that are not OCI artifacts and 404 if mirrored. The skill can't read component_type at runtime (versions.yaml, which carries it, isn't shipped beside manifest.yaml), so this is a hand-maintained list. Default nvcf-stack is component_type: package upstream (an SBOM umbrella label).
REGSYNC_PARALLEL	4	Parallel copy count in generated regsync config.
DRY_RUN	false	Generate config, repo list, and inventory without creating repos or copying.

Never print NGC API keys, ECR passwords, OpenBao tokens, kubeconfig contents, or other secret values. The generated regsync file uses environment placeholders only: {{env "NGC_API_KEY"}} and {{env "ECR_PASSWORD"}}.

Source Registry Resolution

Use image-manifest/source-registry-config.yaml as the primary resolver. The mirror workflow maps manifest groups to source-registry plugins with the same mapping as the Azure ACR mirror flow:

Manifest group	Plugin key
nvcf-self-managed-stack	nvcf_self_managed_stack
nvcf-compute-plane-stack	nvcf_compute_plane_stack
container_cache	container_cache
gxcache	gxcache
ucc	ucc
ddcs	ddcs
lls	lls
storage-api	storage_service, then storage_api

If the config file exists, do not use stale hardcoded defaults. If the config file is missing, fall back only to documented environment overrides:

export NGC_SOURCE_REGISTRY_nvcf_self_managed_stack="nvcr.io/nvidia/nvcf"
export NGC_SOURCE_REGISTRY_storage_api="nvcr.io/nvidia/omniverse"

The legacy NGC_SOURCE_REPO_<group> form is also accepted only in this missing config fallback mode, using NGC_SOURCE_REGISTRY as the host and nvcr.io as the default host.

Mirror Workflow

Run from a checkout that has image-manifest/manifest.yaml, or set MANIFEST_PATH and SOURCE_REGISTRY_CONFIG explicitly.

The generated mirror script must write:

• generated/ecr-regsync.yaml
• generated/ecr-repositories.txt
• generated/ecr-inventory.tsv

Each sync entry must use:

source: <source_registry>/<image>:<tag>
target: <aws_account>.dkr.ecr.<region>.amazonaws.com/<TARGET_REPOSITORY>/<image>:<tag>

The same rule applies to Helm OCI charts, where <image> is the helm-* chart artifact name.

Some plugins (ucc, ddcs, storage_api) set helm_repo_url in source-registry-config.yaml: their charts are served over an HTTP Helm repo, not as OCI artifacts under source_registry. For those groups the chart name drops the helm- prefix and the chart is helm pulled over HTTP and helm pushed to ECR, instead of being OCI-copied via regsync.

set -euo pipefail

for tool in yq; do
  command -v "$tool" >/dev/null 2>&1 || {
    echo "ERROR: $tool not installed" >&2
    exit 1
  }
done

: "${AWS_ACCOUNT_ID:?Set AWS_ACCOUNT_ID}"
: "${AWS_REGION:?Set AWS_REGION}"
: "${CLUSTER_NAME:?Set CLUSTER_NAME}"
TARGET_REPOSITORY="${TARGET_REPOSITORY:-$CLUSTER_NAME}"
if [[ -z "$TARGET_REPOSITORY" ]]; then
  echo "ERROR: TARGET_REPOSITORY is empty. Set it or CLUSTER_NAME." >&2
  exit 1
fi

REPO_ROOT="${REPO_ROOT:-$(pwd)}"
MANIFEST_PATH="${MANIFEST_PATH:-$REPO_ROOT/image-manifest/manifest.yaml}"
SOURCE_REGISTRY_CONFIG="${SOURCE_REGISTRY_CONFIG:-$(dirname "$MANIFEST_PATH")/source-registry-config.yaml}"
INCLUDE_HELM_CHARTS="${INCLUDE_HELM_CHARTS:-true}"
INCLUDE_GROUPS="${INCLUDE_GROUPS:-}"
REGSYNC_PARALLEL="${REGSYNC_PARALLEL:-4}"
DRY_RUN="${DRY_RUN:-false}"
# Manifest keys to skip BY NAME because they are not OCI artifacts and 404 if
# mirrored. This is a hand-maintained deny-list, NOT a type lookup: the skill
# can't see component_type at runtime — versions.yaml (which carries it) is not




SKIP_KEYS=
IFS=  -ra SKIP_KEYS_ARR <<< 

 -p 
CONFIG=
REPOS=
INVENTORY=
HELM_HTTP=
ECR_REGISTRY=

 [[ ! -f  ]]; 
    >&2
   1


() {
    
    nvcf-self-managed-stack)   nvcf_self_managed_stack ;;
    nvcf-compute-plane-stack)   nvcf_compute_plane_stack ;;
    container_cache)   container_cache ;;
    gxcache)   gxcache ;;
    ucc)   ucc ;;
    ddcs)   ddcs ;;
    lls)   lls ;;
    storage-api)   storage_service storage_api ;;
  
}

() {
     |  -c  
}

() {
   group= plugin value var
   [[ -f  ]]; 
     IFS=  -r plugin; 
      value=
       [[ -n  &&  !=  ]]; 
          
         0
      
     < <(plugin_for_group )
     1
  

  var=
  value=
   [[ -n  ]]; 
      
     0
  

  var=
  value=
   [[ -n  ]]; 
       
     0
  
   1
}






() {
   group= plugin value
  [[ -f  ]] ||  0
   IFS=  -r plugin; 
    value=
     [[ -n  &&  !=  ]]; 
        
       0
    
   < <(plugin_for_group )
}

 >  <<
: > 
: > 
: > 

 -t ALL_GROUPS < <(
  yq -r  
)





 [[ -n  ]]; 
  IFS=  -ra REQUESTED <<< 
  SELECTED_GROUPS=()
   group  ; 
     requested  ; 
      [[  ==  ]] && SELECTED_GROUPS+=() && 
    
  

  SELECTED_GROUPS=()


 group  ; 
  source_registry= || {
      >&2
     1
  }
  helm_repo_url=

  
  
  
   IFS=  -r pair; 
    name=
    tag=
    skip=
     s  ;  [[  ==  ]] && skip=1 && ; 
     [[ -n  ]]; 
        >&2
      
    
     [[  == helm-* ]];  kind=;  kind=; 
    [[  ==  &&  !=  ]] && 

    
    
    
     [[  ==  && -n  ]]; 
      chart=
      repo=
         >> 
        \
               >> 
            >> 
      
    

    repo=
    =
    target=

       >> 
      \
             >> 
     >>  <<
   < <(yq -r  )


 -u  -o 

 [[  ==  ]]; 
   
  sed  
   [[ -s  ]]; 
     IFS=$  -r group chart tag repo; 
       
     < 
  
   0


 tool  aws regsync; 
   -v  >/dev/null 2>&1 || {
      >&2
     1
  }

 [[ -s  ]]; 
   -v helm >/dev/null 2>&1 || {
      >&2
     1
  }


: 

ACTIVE_ACCOUNT=
 [[  !=  ]]; 
    >&2
   1


 IFS=  -r repo; 
  [[ -z  ]] && 
   aws ecr describe-repositories \
      --region  \
      --repository-names  >/dev/null 2>&1; 
    
  
  aws ecr create-repository \
    --region  \
    --repository-name  \
    --image-scanning-configuration scanOnPush= \
    --encryption-configuration encryptionType=AES256 \
    >/dev/null
 < 

 ECR_PASSWORD
ECR_PASSWORD=
regsync once -c 



 [[ -s  ]]; 
   
    | helm registry login --username AWS --password-stdin 
  CHARTS_DIR=
    EXIT
   -A HELM_REPO_ADDED=()
   IFS=$  -r group chart tag repo; 
    helm_repo_url=
    [[ -z  ]] && 
    =
     [[ -z  ]]; 
      helm repo add   \
        --username  --password  >/dev/null
      HELM_REPO_ADDED[]=1
    
    helm pull  --version  -d 
    helm push  
     -f 
   < 

 ECR_PASSWORD

This creates missing ECR repositories before mirroring:

${TARGET_REPOSITORY}/${image}
${TARGET_REPOSITORY}/${chart}

NVCF Helmfile Handoff

After mirroring, configure the NVCF Helmfile environment to pull both images and OCI charts from ECR:

global:
  image:
    registry: <aws_account>.dkr.ecr.<region>.amazonaws.com
    repository: <TARGET_REPOSITORY>
  helm:
    sources:
      registry: <aws_account>.dkr.ecr.<region>.amazonaws.com
      repository: <TARGET_REPOSITORY>

For EKS pulls, confirm the node role or pod identity has ECR read access:

ecr:GetAuthorizationToken
ecr:BatchCheckLayerAvailability
ecr:GetDownloadUrlForLayer
ecr:BatchGetImage

Verification

The mirror script writes generated/ecr-inventory.tsv with one row per expected manifest tag. After mirroring, verify every image and chart tag exists in ECR:

missing=0
while IFS=$'\t' read -r kind group name tag repo source target; do
  if ! aws ecr describe-images \
      --region "$AWS_REGION" \
      --repository-name "$repo" \
      --image-ids "imageTag=$tag" >/dev/null 2>&1; then
    echo "MISSING: $repo:$tag ($kind $group/$name)"
    missing=1
  fi
done < generated/ecr-inventory.tsv
test "$missing" -eq 0

For a local dry-run count check against the current manifest:

MANIFEST_PATH=image-manifest/manifest.yaml
IMAGE_COUNT=$(yq -r '
  to_entries | .[]
  | select(.key != "version" and .key != "generated_at")
  | .value | to_entries | .[]
  | select(.key | test("^helm-") | not)
  | (.value | if type == "array" then .[] else . end)
  | "x"
' "$MANIFEST_PATH" | wc -l | tr -d ' ')
CHART_COUNT=$(yq -r '
  to_entries | .[]
  | select(.key != "version" and .key != "generated_at")
  | .value | to_entries | .[]
  | select(.key | test("^helm-"))
  | (.value | if type == "array" then .[] else . end)
  | "x"
' "$MANIFEST_PATH" | wc -l | tr -d ' ')
echo "images=$IMAGE_COUNT charts=$CHART_COUNT"

Expected for the referenced manifest:

images=42 charts=20

Troubleshooting

Symptom	Cause	Fix
Generated entry count is lower than manifest count	Source registry config is missing or a group cannot resolve	Provide image-manifest/source-registry-config.yaml; only use env fallbacks when that file is unavailable.
TARGET_REPOSITORY is empty	Both TARGET_REPOSITORY and CLUSTER_NAME are unset	Stop and set one; do not mirror to the registry root.
Image pull fails after a successful mirror	Chart-rendered image name differs from manifest name or the source registry content drifted	Compare chart render output with generated/ecr-inventory.tsv, then mirror the exact rendered image or update the manifest/source mapping.

Live deployment drift observed:

• The chart/manifests can drift from available registry content.
• nvct-api rendered nvct-service-oss:1.3.11 from staging, but that tag was not pullable there; it existed in ncp-dev and in ECR as ov-qa-cluster/nvct-service-oss:1.3.11.
• nvcf-api rendered nvcf-service-oss:2.234.0, but the manifest image name is strap:2.234.0; strap:2.234.0 was pullable from ncp-dev but missing from ECR.
• nats-auth-callout-service:0.3.3 existed in ECR as ov-qa-cluster/nvcf-nats-auth-callout-service:0.3.3.

Do not hardcode ov-qa-cluster; use ${TARGET_REPOSITORY}. The observed names are only examples for diagnosing chart/render drift.

Validation Checklist

• Mirror workflow consumes image-manifest/manifest.yaml as source of truth.
• Mirror workflow resolves source registries from source-registry-config.yaml.
• Scalar and list-valued tags both produce sync entries.
• Non-helm image entries are all present in generated regsync config.
• helm-* chart entries are present when INCLUDE_HELM_CHARTS=true.
• Missing ECR repositories are created before regsync once.
• No secret values are printed or written into generated files.
• Verification checks every generated inventory row in ECR.