| name | ecr-mirror |
| description | Mirror every KAS/NVCF manifest image and optional Helm OCI chart from NGC into AWS ECR using generated regsync config. |
| version | 1.1.0 |
| author | NVIDIA Omniverse Streaming |
| tags | ["aws","ecr","regsync","nvcf"] |
| tools | ["Shell","Read","Write"] |
ECR Mirror
Purpose
Populate AWS ECR from image-manifest/manifest.yaml before installing the
NVCF self-managed stack on EKS. The manifest is the source of truth: mirror
every non-helm image:tag entry, and mirror helm-* OCI chart artifacts when
INCLUDE_HELM_CHARTS=true. For AWS this defaults to true.
Use this after AWS infra has created or authorized ECR access. Do not use this
skill for Terraform, EKS cluster creation, NVCF installation, function deploys,
or stream validation.
Inputs
Required:
| Variable | Description |
|---|
AWS_ACCOUNT_ID | AWS account that owns the target ECR registry. |
AWS_REGION | Target ECR region, for example us-west-2. |
CLUSTER_NAME | Default ECR repository prefix when TARGET_REPOSITORY is unset. |
Optional:
| Variable | Default | Description |
|---|
NGC_API_KEY | unset | Required only when DRY_RUN=false; export in the shell for regsync and never print it. |
TARGET_REPOSITORY | ${CLUSTER_NAME} | Non-empty ECR repository prefix. |
MANIFEST_PATH | ${REPO_ROOT}/image-manifest/manifest.yaml | Manifest source of truth. |
SOURCE_REGISTRY_CONFIG | beside MANIFEST_PATH | Source registry resolver. |
INCLUDE_HELM_CHARTS | true | Include helm-* OCI chart artifacts in the regsync config. |
INCLUDE_GROUPS | empty | Comma-separated manifest groups to mirror; empty means all groups. |
SKIP_KEYS | nvcf-stack | Comma-separated manifest keys to skip by name — entries that are not OCI artifacts and 404 if mirrored. The skill can't read component_type at runtime (versions.yaml, which carries it, isn't shipped beside manifest.yaml), so this is a hand-maintained list. Default nvcf-stack is component_type: package upstream (an SBOM umbrella label). |
REGSYNC_PARALLEL | 4 | Parallel copy count in generated regsync config. |
DRY_RUN | false | Generate config, repo list, and inventory without creating repos or copying. |
Never print NGC API keys, ECR passwords, OpenBao tokens, kubeconfig contents, or
other secret values. The generated regsync file uses environment placeholders
only: {{env "NGC_API_KEY"}} and {{env "ECR_PASSWORD"}}.
Source Registry Resolution
Use image-manifest/source-registry-config.yaml as the primary resolver. The
mirror workflow maps manifest groups to source-registry plugins with the same
mapping as the Azure ACR mirror flow:
| Manifest group | Plugin key |
|---|
nvcf-self-managed-stack | nvcf_self_managed_stack |
nvcf-compute-plane-stack | nvcf_compute_plane_stack |
container_cache | container_cache |
gxcache | gxcache |
ucc | ucc |
ddcs | ddcs |
lls | lls |
storage-api | storage_service, then storage_api |
If the config file exists, do not use stale hardcoded defaults. If the config
file is missing, fall back only to documented environment overrides:
export NGC_SOURCE_REGISTRY_nvcf_self_managed_stack="nvcr.io/nvidia/nvcf"
export NGC_SOURCE_REGISTRY_storage_api="nvcr.io/nvidia/omniverse"
The legacy NGC_SOURCE_REPO_<group> form is also accepted only in this missing
config fallback mode, using NGC_SOURCE_REGISTRY as the host and nvcr.io as
the default host.
Mirror Workflow
Run from a checkout that has image-manifest/manifest.yaml, or set
MANIFEST_PATH and SOURCE_REGISTRY_CONFIG explicitly.
The generated mirror script must write:
generated/ecr-regsync.yamlgenerated/ecr-repositories.txtgenerated/ecr-inventory.tsv
Each sync entry must use:
source: <source_registry>/<image>:<tag>
target: <aws_account>.dkr.ecr.<region>.amazonaws.com/<TARGET_REPOSITORY>/<image>:<tag>
The same rule applies to Helm OCI charts, where <image> is the helm-* chart
artifact name.
Some plugins (ucc, ddcs, storage_api) set helm_repo_url in
source-registry-config.yaml: their charts are served over an HTTP Helm repo,
not as OCI artifacts under source_registry. For those groups the chart name
drops the helm- prefix and the chart is helm pulled over HTTP and
helm pushed to ECR, instead of being OCI-copied via regsync.
set -euo pipefail
for tool in yq; do
command -v "$tool" >/dev/null 2>&1 || {
echo "ERROR: $tool not installed" >&2
exit 1
}
done
: "${AWS_ACCOUNT_ID:?Set AWS_ACCOUNT_ID}"
: "${AWS_REGION:?Set AWS_REGION}"
: "${CLUSTER_NAME:?Set CLUSTER_NAME}"
TARGET_REPOSITORY="${TARGET_REPOSITORY:-$CLUSTER_NAME}"
if [[ -z "$TARGET_REPOSITORY" ]]; then
echo "ERROR: TARGET_REPOSITORY is empty. Set it or CLUSTER_NAME." >&2
exit 1
fi
REPO_ROOT="${REPO_ROOT:-$(pwd)}"
MANIFEST_PATH="${MANIFEST_PATH:-$REPO_ROOT/image-manifest/manifest.yaml}"
SOURCE_REGISTRY_CONFIG="${SOURCE_REGISTRY_CONFIG:-$(dirname "$MANIFEST_PATH")/source-registry-config.yaml}"
INCLUDE_HELM_CHARTS="${INCLUDE_HELM_CHARTS:-true}"
INCLUDE_GROUPS="${INCLUDE_GROUPS:-}"
REGSYNC_PARALLEL="${REGSYNC_PARALLEL:-4}"
DRY_RUN="${DRY_RUN:-false}"
SKIP_KEYS="${SKIP_KEYS:-nvcf-stack}"
IFS=',' read -ra SKIP_KEYS_ARR <<< "$SKIP_KEYS"
mkdir -p "$REPO_ROOT/generated"
CONFIG="$REPO_ROOT/generated/ecr-regsync.yaml"
REPOS="$REPO_ROOT/generated/ecr-repositories.txt"
INVENTORY="$REPO_ROOT/generated/ecr-inventory.tsv"
HELM_HTTP="$REPO_ROOT/generated/ecr-helm-http.tsv"
ECR_REGISTRY="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com"
if [[ ! -f "$MANIFEST_PATH" ]]; then
echo "ERROR: manifest not found at $MANIFEST_PATH" >&2
exit 1
fi
plugin_for_group() {
case "$1" in
nvcf-self-managed-stack) printf '%s\n' nvcf_self_managed_stack ;;
nvcf-compute-plane-stack) printf '%s\n' nvcf_compute_plane_stack ;;
container_cache) printf '%s\n' container_cache ;;
gxcache) printf '%s\n' gxcache ;;
ucc) printf '%s\n' ucc ;;
ddcs) printf '%s\n' ddcs ;;
lls) printf '%s\n' lls ;;
storage-api) printf '%s\n' storage_service storage_api ;;
esac
}
env_name() {
printf '%s' "$1" | tr -c '[:alnum:]' '_'
}
source_registry_for_group() {
local group="$1" plugin value var
if [[ -f "$SOURCE_REGISTRY_CONFIG" ]]; then
while IFS= read -r plugin; do
value="$(yq -r ".upstream_plugins.\"$plugin\".source_registry // \"\"" "$SOURCE_REGISTRY_CONFIG")"
if [[ -n "$value" && "$value" != "null" ]]; then
printf '%s' "${value%/}"
return 0
fi
done < <(plugin_for_group "$group")
return 1
fi
var="NGC_SOURCE_REGISTRY_$(env_name "$group")"
value="${!var:-}"
if [[ -n "$value" ]]; then
printf '%s' "${value%/}"
return 0
fi
var="NGC_SOURCE_REPO_$(env_name "$group")"
value="${!var:-}"
if [[ -n "$value" ]]; then
printf '%s/%s' "${NGC_SOURCE_REGISTRY:-nvcr.io}" "${value#/}"
return 0
fi
return 1
}
helm_repo_url_for_group() {
local group="$1" plugin value
[[ -f "$SOURCE_REGISTRY_CONFIG" ]] || return 0
while IFS= read -r plugin; do
value="$(yq -r ".upstream_plugins.\"$plugin\".helm_repo_url // \"\"" "$SOURCE_REGISTRY_CONFIG")"
if [[ -n "$value" && "$value" != "null" ]]; then
printf '%s' "${value%/}"
return 0
fi
done < <(plugin_for_group "$group")
}
cat > "$CONFIG" <<EOF
version: 1
creds:
- registry: nvcr.io
user: \$oauthtoken
pass: '{{env "NGC_API_KEY"}}'
repoAuth: true
- registry: ${ECR_REGISTRY}
user: AWS
pass: '{{env "ECR_PASSWORD"}}'
defaults:
parallel: ${REGSYNC_PARALLEL}
sync:
EOF
: > "$REPOS"
: > "$INVENTORY"
: > "$HELM_HTTP"
mapfile -t ALL_GROUPS < <(
yq -r 'keys | .[] | select(. != "version" and . != "generated_at")' "$MANIFEST_PATH"
)
if [[ -n "$INCLUDE_GROUPS" ]]; then
IFS=',' read -ra REQUESTED <<< "$INCLUDE_GROUPS"
SELECTED_GROUPS=()
for group in "${ALL_GROUPS[@]}"; do
for requested in "${REQUESTED[@]}"; do
[[ "$group" == "$requested" ]] && SELECTED_GROUPS+=("$group") && break
done
done
else
SELECTED_GROUPS=("${ALL_GROUPS[@]}")
fi
for group in "${SELECTED_GROUPS[@]}"; do
source_registry="$(source_registry_for_group "$group")" || {
echo "ERROR: no source registry for manifest group $group" >&2
exit 1
}
helm_repo_url="$(helm_repo_url_for_group "$group")"
while IFS= read -r pair; do
name="${pair%%:*}"
tag="${pair#*:}"
skip=""
for s in "${SKIP_KEYS_ARR[@]}"; do [[ "$name" == "$s" ]] && skip=1 && break; done
if [[ -n "$skip" ]]; then
echo "SKIP: $group/$name (non-OCI / package artifact, not mirrorable)" >&2
continue
fi
if [[ "$name" == helm-* ]]; then kind="helm"; else kind="image"; fi
[[ "$kind" == "helm" && "$INCLUDE_HELM_CHARTS" != "true" ]] && continue
if [[ "$kind" == "helm" && -n "$helm_repo_url" ]]; then
chart="${name#helm-}"
repo="${TARGET_REPOSITORY}/${chart}"
printf '%s\n' "$repo" >> "$REPOS"
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
"helm-http" "$group" "$chart" "$tag" "$repo" "${helm_repo_url}/${chart}" "${ECR_REGISTRY}/${repo}" >> "$INVENTORY"
printf '%s\t%s\t%s\t%s\n' "$group" "$chart" "$tag" "$repo" >> "$HELM_HTTP"
continue
fi
repo="${TARGET_REPOSITORY}/${name}"
source="${source_registry}/${name}:${tag}"
target="${ECR_REGISTRY}/${repo}:${tag}"
printf '%s\n' "$repo" >> "$REPOS"
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
"$kind" "$group" "$name" "$tag" "$repo" "$source" "$target" >> "$INVENTORY"
cat >> "$CONFIG" <<EOF
- source: ${source}
target: ${target}
type: image
EOF
done < <(yq -r "
.\"$group\"
| to_entries
| .[]
| .key as \$name
| (.value | [.] | flatten | .[]) as \$tag
| \$name + \":\" + \$tag
" "$MANIFEST_PATH")
done
sort -u "$REPOS" -o "$REPOS"
if [[ "$DRY_RUN" == "true" ]]; then
echo "DRY_RUN=true: generated $CONFIG, $REPOS, $INVENTORY, and $HELM_HTTP"
sed 's/^/DRY: ensure ECR repo /' "$REPOS"
if [[ -s "$HELM_HTTP" ]]; then
while IFS=$'\t' read -r group chart tag repo; do
echo "DRY: [$group] helm pull $(helm_repo_url_for_group "$group")/${chart}:${tag} (HTTP) -> push oci://${ECR_REGISTRY}/${repo}"
done < "$HELM_HTTP"
fi
exit 0
fi
for tool in aws regsync; do
command -v "$tool" >/dev/null 2>&1 || {
echo "ERROR: $tool not installed" >&2
exit 1
}
done
if [[ -s "$HELM_HTTP" ]]; then
command -v helm >/dev/null 2>&1 || {
echo "ERROR: helm not installed (required to mirror HTTP helm charts)" >&2
exit 1
}
fi
: "${NGC_API_KEY:?Set NGC_API_KEY in the shell without printing it}"
ACTIVE_ACCOUNT="$(aws sts get-caller-identity --query Account --output text)"
if [[ "$ACTIVE_ACCOUNT" != "$AWS_ACCOUNT_ID" ]]; then
echo "ERROR: active AWS account is $ACTIVE_ACCOUNT, expected $AWS_ACCOUNT_ID" >&2
exit 1
fi
while IFS= read -r repo; do
[[ -z "$repo" ]] && continue
if aws ecr describe-repositories \
--region "$AWS_REGION" \
--repository-names "$repo" >/dev/null 2>&1; then
continue
fi
aws ecr create-repository \
--region "$AWS_REGION" \
--repository-name "$repo" \
--image-scanning-configuration scanOnPush=true \
--encryption-configuration encryptionType=AES256 \
>/dev/null
done < "$REPOS"
export ECR_PASSWORD
ECR_PASSWORD="$(aws ecr get-login-password --region "$AWS_REGION")"
regsync once -c "$CONFIG"
if [[ -s "$HELM_HTTP" ]]; then
echo "--- Mirroring HTTP helm charts ---"
echo "$ECR_PASSWORD" | helm registry login --username AWS --password-stdin "$ECR_REGISTRY"
CHARTS_DIR="$(mktemp -d -t ecr-mirror-charts.XXXXXX)"
trap 'rm -rf "$CHARTS_DIR"' EXIT
declare -A HELM_REPO_ADDED=()
while IFS=$'\t' read -r group chart tag repo; do
helm_repo_url="$(helm_repo_url_for_group "$group")"
[[ -z "$helm_repo_url" ]] && continue
alias="src_$(env_name "$group")"
if [[ -z "${HELM_REPO_ADDED[$alias]:-}" ]]; then
helm repo add "$alias" "$helm_repo_url" \
--username '$oauthtoken' --password "$NGC_API_KEY" >/dev/null
HELM_REPO_ADDED[$alias]=1
fi
helm pull "$alias/$chart" --version "$tag" -d "$CHARTS_DIR"
helm push "$CHARTS_DIR/${chart}-${tag}.tgz" "oci://${ECR_REGISTRY}/${TARGET_REPOSITORY}"
rm -f "$CHARTS_DIR/${chart}-${tag}.tgz"
done < "$HELM_HTTP"
fi
unset ECR_PASSWORD
This creates missing ECR repositories before mirroring:
${TARGET_REPOSITORY}/${image}
${TARGET_REPOSITORY}/${chart}
NVCF Helmfile Handoff
After mirroring, configure the NVCF Helmfile environment to pull both images and
OCI charts from ECR:
global:
image:
registry: <aws_account>.dkr.ecr.<region>.amazonaws.com
repository: <TARGET_REPOSITORY>
helm:
sources:
registry: <aws_account>.dkr.ecr.<region>.amazonaws.com
repository: <TARGET_REPOSITORY>
For EKS pulls, confirm the node role or pod identity has ECR read access:
ecr:GetAuthorizationToken
ecr:BatchCheckLayerAvailability
ecr:GetDownloadUrlForLayer
ecr:BatchGetImage
Verification
The mirror script writes generated/ecr-inventory.tsv with one row per
expected manifest tag. After mirroring, verify every image and chart tag exists
in ECR:
missing=0
while IFS=$'\t' read -r kind group name tag repo source target; do
if ! aws ecr describe-images \
--region "$AWS_REGION" \
--repository-name "$repo" \
--image-ids "imageTag=$tag" >/dev/null 2>&1; then
echo "MISSING: $repo:$tag ($kind $group/$name)"
missing=1
fi
done < generated/ecr-inventory.tsv
test "$missing" -eq 0
For a local dry-run count check against the current manifest:
MANIFEST_PATH=image-manifest/manifest.yaml
IMAGE_COUNT=$(yq -r '
to_entries | .[]
| select(.key != "version" and .key != "generated_at")
| .value | to_entries | .[]
| select(.key | test("^helm-") | not)
| (.value | if type == "array" then .[] else . end)
| "x"
' "$MANIFEST_PATH" | wc -l | tr -d ' ')
CHART_COUNT=$(yq -r '
to_entries | .[]
| select(.key != "version" and .key != "generated_at")
| .value | to_entries | .[]
| select(.key | test("^helm-"))
| (.value | if type == "array" then .[] else . end)
| "x"
' "$MANIFEST_PATH" | wc -l | tr -d ' ')
echo "images=$IMAGE_COUNT charts=$CHART_COUNT"
Expected for the referenced manifest:
images=42 charts=20
Troubleshooting
| Symptom | Cause | Fix |
|---|
| Generated entry count is lower than manifest count | Source registry config is missing or a group cannot resolve | Provide image-manifest/source-registry-config.yaml; only use env fallbacks when that file is unavailable. |
TARGET_REPOSITORY is empty | Both TARGET_REPOSITORY and CLUSTER_NAME are unset | Stop and set one; do not mirror to the registry root. |
| Image pull fails after a successful mirror | Chart-rendered image name differs from manifest name or the source registry content drifted | Compare chart render output with generated/ecr-inventory.tsv, then mirror the exact rendered image or update the manifest/source mapping. |
Live deployment drift observed:
- The chart/manifests can drift from available registry content.
nvct-api rendered nvct-service-oss:1.3.11 from staging, but that tag was
not pullable there; it existed in ncp-dev and in ECR as
ov-qa-cluster/nvct-service-oss:1.3.11.nvcf-api rendered nvcf-service-oss:2.234.0, but the manifest image name
is strap:2.234.0; strap:2.234.0 was pullable from ncp-dev but missing
from ECR.nats-auth-callout-service:0.3.3 existed in ECR as
ov-qa-cluster/nvcf-nats-auth-callout-service:0.3.3.
Do not hardcode ov-qa-cluster; use ${TARGET_REPOSITORY}. The observed names
are only examples for diagnosing chart/render drift.
Validation Checklist
