Prepare mirrors and transfer artifacts, configure DeepOps, deploy Slurm or Kubernetes GPU clusters without Internet access, and validate them with machine-readable gates. Use for disconnected, restricted-egress, offline, or air-gapped DeepOps installations and for diagnosing missing package, file, chart, or container artifacts.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Instruções da origem · Visualização somente leitura
name
deploy-airgapped
description
Prepare mirrors and transfer artifacts, configure DeepOps, deploy Slurm or Kubernetes GPU clusters without Internet access, and validate them with machine-readable gates. Use for disconnected, restricted-egress, offline, or air-gapped DeepOps installations and for diagnosing missing package, file, chart, or container artifacts.
Deploy DeepOps in an air-gapped environment
Preconditions and boundaries
Use an Internet-connected staging machine and a separate provisioning machine
that can reach every target and every internal mirror. Do not connect an
isolated network to the Internet for this procedure.
Fix one DeepOps commit, target OS release, CPU architecture, cluster type,
inventory, and enabled feature set before mirroring. Repository and image
requirements change when any of these inputs change.
Provide complete distribution package mirrors plus internal HTTP package/file
service and container registry endpoints. Use trusted TLS where possible; if
the registry is intentionally insecure, configure it explicitly.
Bootstrap the isolated provisioning machine with the OS packages and Python
environment required by scripts/setup.sh before closing the boundary, or
provide approved internal OS and Python package indexes. Copying the source
tree alone does not install Ansible.
Reserve enough storage for repository metadata, packages, container images,
charts, the initialized DeepOps checkout, and two copies during transfer.
Confirm that every target is fully manageable and has no active users or
workloads. Driver installation can reboot hosts.
Run DeepOps commands from the repository root. Keep secrets out of archives
and store site configuration in config/, never config.example/.
DeepOps does not provide supported mirror-building automation. Do not run
scripts/airgap/build_offline_cache.sh: it calls the removed
playbooks/airgap/build-offline-cache.yml and cannot complete in this tree.
Build and verify the mirrors explicitly as described below.
1. Freeze the deployment inputs
Record the DeepOps commit and initialize every submodule on the connected
staging machine:
git rev-parse HEAD
git submodule update --init --recursive
./scripts/setup.sh
Copy config.example/ to config/. Define the real hosts in
config/inventory before collecting artifacts so the selected cluster path
and optional components are known:
Slurm: controller/login hosts in [slurm-master], compute hosts in
[slurm-node]; a single host may be in both groups.
Kubernetes: control-plane hosts in [kube_control_plane] and [etcd],
workers in [kube_node]; a single host may be in all three groups.
Review the resolved inventory. Stop if it contains any unintended host:
ansible-inventory -i config/inventory --list
Record the target OS release/architecture and all enabled roles. Mirror
every dependency of that exact profile. In particular, account for:
distribution, Docker CE, NVIDIA CUDA/driver, NVIDIA Container Toolkit,
and EPEL repositories where applicable;
direct-download archives referenced by enabled role defaults;
every container image and Helm chart used by the chosen Slurm or
Kubernetes path;
the initialized Kubespray submodule, Galaxy roles/collections, and Python
packages needed on the provisioning machine.
2. Build package mirrors on the connected side
Ubuntu/APT
Install apt-mirror, set base_path in /etc/apt/mirror.list (for example,
/var/repos), and add the suites for the target release. For Ubuntu 24.04
(noble), the current DeepOps documentation uses these sources:
deb http://archive.ubuntu.com/ubuntu noble main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu noble-security main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu noble-updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu noble-backports main restricted universe multiverse
deb https://download.docker.com/linux/ubuntu noble stable
deb https://nvidia.github.io/libnvidia-container/stable/deb/amd64 /
deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64 /
Add DGX OS or MAAS repositories only when the selected profile uses them. Use
the matching suite and architecture instead of noble/amd64 for another
target. Download exactly the configured set:
sudomkdir -p /var/repos
sudo apt-mirror
Download the signing material that DeepOps fetches directly, plus the RPM
bootstrap files when building an Enterprise Linux mirror. Keep these outside
mirror/ so their archive paths are deterministic:
Replace ubuntu2404, rhel9, and 9 together for another supported target.
Verify the downloaded fingerprints against the vendor's published values under
the site's package-trust process before transfer; do not turn off GPG checking.
Enterprise Linux/RPM
Configure the target distribution repositories plus EPEL, Docker CE, CUDA,
libnvidia-container, and nvidia-container-toolkit. For EL9, download the
checked-in documentation's complete additional-repository set into the
top-level repo-ID directories created by reposync:
for repo_id in epel docker-ce-stable cuda-rhel9-x86_64 \
libnvidia-container nvidia-container-toolkit; dosudo reposync -l --repoid="${repo_id}" --downloadcomps \
--download-metadata --download_path=/var/repos
sudo createrepo "/var/repos/${repo_id}"done
Use cuda-rhel8-x86_64 instead on EL8. Compare the final IDs and package
contents with /etc/yum.repos.d/ and the enabled workload on a connected
reference host; distribution repositories remain a separate prerequisite.
3. Mirror direct files, charts, and images
Download the direct files selected by the current default Slurm profile.
These names and versions come from the checked-in role defaults:
Enroot is also enabled by default, but its package filenames are owned by
the pinned nvidia.enroot Galaxy role rather than this repository. After
./scripts/setup.sh, use that installed role's real package list to
download the target-OS packages and generate the matching Ansible override.
Choose enroot_deb_packages for Ubuntu or enroot_rpm_packages for
Enterprise Linux; repeat for both when the inventory mixes OS families:
Add the generated Enroot YAML list to config/group_vars/all.yml. On
Enterprise Linux, also override epel_package and epel_key_url; several
default roles import the EPEL key directly, so mirroring packages alone is
insufficient.
DCGM needs no separate file mirror: the nvidia_dcgm role installs
dcgm_pkg_name (datacenter-gpu-manager) from the CUDA package
repository, so it is covered by the CUDA repository mirror below — verify
the DCGM package is present in the mirrored CUDA repo content.
The default Slurm configuration also enables CUDA installation, whose
Ubuntu and Enterprise Linux tasks fetch the CUDA repository and its
signing material from developer.download.nvidia.com. Mirroring the CUDA
packages alone does not redirect these tasks. On Ubuntu, the role uses
nvidia_driver_ubuntu_cuda_keyring_url only to install the keyring package;
it does not use nvidia_driver_ubuntu_cuda_repo_baseurl to write an APT
source. The keyring package writes an upstream source that must be replaced
on every target as shown in step 5. On Enterprise Linux, override both
repository variables after mirroring the metadata and GPG key:
nvidia_driver_ubuntu_cuda_keyring_url (Ubuntu, plus the target-side APT
source replacement in step 5)
nvidia_driver_rhel_cuda_repo_baseurl and
nvidia_driver_rhel_cuda_repo_gpgkey (Enterprise Linux)
Either mirror and override every enabled dependency or explicitly disable
the optional component. Do not silently fall back to its public URL.
Mirror the Helm repositories/charts required by the selected Kubernetes
profile. The default tree references the stable Helm repository, GPU
Operator v26.3.3, and, when enabled, NFS subdir external provisioner
4.0.18. Set gpu_operator_helm_repo and
k8s_nfs_client_helm_repo to internal chart repositories.
Archive the chart versions selected by current DeepOps defaults on the
connected side:
Omit the NFS chart only when k8s_nfs_client_provisioner: false. Publish
/tmp/charts from an internal HTTP service and set the two repository
variables to that URL.
Generate and collect the exact Kubespray file and image set from the pinned
submodule. generate_list.yml runs only against localhost; passing
-i config/inventory alone does not apply variables from DeepOps'
k8s_cluster group. Export the effective variables for one representative
Kubernetes host, review the selected versions, and pass that JSON as
extra-vars so the localhost play uses the deployment's resolved overrides.
Kubernetes download versions must be cluster-wide; stop if control-plane and
worker hosts resolve different artifact-selection variables. Then render the
enabled DeepOps charts with the same values used by their roles and append
their images to Kubespray's list before creating the image archive:
task_deepops_root=$(pwd)
task_offline_dir="${task_deepops_root}/submodules/kubespray/contrib/offline"
task_kube_host=kube01 # replace with one real kube_control_plane or kube_node host
ansible-inventory -i config/inventory --host "${task_kube_host}" \
> /tmp/kubespray-effective-vars.json
python3 - /tmp/kubespray-effective-vars.json <<'PY'
import json
import sys
variables = json.load(open(sys.argv[1], encoding="utf-8"))
for name in ("kube_version", "container_manager", "etcd_deployment_type"):
print(f"{name}={variables.get(name, '<Kubespray default>')}")
PY
cd"${task_offline_dir}"
./generate_list.sh -i "${task_deepops_root}/config/inventory" \
-e "@/tmp/kubespray-effective-vars.json"test -s temp/files.list
test -s temp/images.list
NO_HTTP_SERVER=1 ./manage-offline-files.sh
helm template nvidia-gpu-operator \
/tmp/charts/gpu-operator-v26.3.3.tgz \
--namespace gpu-operator \
--set driver.version=580.126.20 \
--set mig.strategy=mixed \
--set driver.enabled= \
-- toolkit.enabled= \
-- dcgm.enabled= \
-- migManager.enabled= \
> /tmp/gpu-operator-rendered.yml
helm template nfs-subdir-external-provisioner \
/tmp/charts/nfs-subdir-external-provisioner-4.0.18.tgz \
> /tmp/nfs-provisioner-rendered.yml
python3 - /tmp/gpu-operator-rendered.yml \
/tmp/nfs-provisioner-rendered.yml << >> temp/images.list
import sys
import yaml
images = ()
def walk(value):
isinstance(value, dict):
repository = value.get()
image = value.get()
version = value.get(, value.get())
isinstance(repository, str) and isinstance(image, str) and version:
reference = f
not reference and not reference.rsplit(, 1)[-1]:
separator = str(version).startswith()
reference = f
images.add(reference)
isinstance(image, str):
= image.rsplit(, 1)[-1]
image or :
images.add(image)
child value.values():
walk(child)
isinstance(value, list):
child value:
walk(child)
manifest_path sys.argv[1:]:
with open(manifest_path, encoding=) as stream:
document yaml.safe_load_all(stream):
walk(document)
not images:
raise SystemExit()
(.(sorted(images)))
PY
-u -o temp/images.list temp/images.list
-z
PRIVATE_REGISTRY=nvcr.io IMAGES_FROM_FILE= \
./manage-offline-container-images.sh create
install -d /tmp/kubespray-offline
offline-files.tar.gz container-images.tar.gz temp/files.list \
temp/images.list /tmp/kubespray-offline/
Pull and archive the CUDA validator plus the images used by enabled Slurm
monitoring/registry roles. For the current validator and registry images:
mkdir -p /tmp/images
docker pull nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
docker save -o /tmp/images/nvidia-cuda-12.4.1-base-ubuntu22.04.tar \
nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
docker pull registry:3.1.1
docker save -o /tmp/images/registry-3.1.1.tar registry:3.1.1
When Slurm monitoring stays enabled, also mirror the images selected by
prometheus_container, grafana_container, alertmanager_container,
node_exporter_container, and nvidia_dcgm_container, then override those
variables with their internal registry paths. The Slurm exporter is a
special case: its default slurm_exporter_build_image: true builds from
golang:1.24 and ubuntu:24.04, and the runtime stage runs apt-get.
Build it while connected, archive it with the other images, and use the
prebuilt image offline:
for image in prom/prometheus:v3.13.0 grafana/grafana:13.1.0 \
prom/alertmanager:v0.33.0 quay.io/prometheus/node-exporter:v1.11.1 \
nvcr.io/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless; do
docker pull "${image}"done
docker save -o /tmp/images/slurm-monitoring.tar \
prom/prometheus:v3.13.0 grafana/grafana:13.1.0 \
prom/alertmanager:v0.33.0 quay.io/prometheus/node-exporter:v1.11.1 \
nvcr.io/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless
docker build -t deepops/prometheus-slurm-exporter:2.0.0 \
roles/prometheus-slurm-exporter/files/docker
docker save -o /tmp/images/prometheus-slurm-exporter-2.0.0.tar \
deepops/prometheus-slurm-exporter:2.0.0
This connected-side build obtains the two base images and runtime APT
packages. Do not leave the default local build enabled in the isolated
environment unless an approved internal build setup supplies both base
images and APT sources.
Archive the initialized checkout separately from site secrets. Keep
.git in the archive: playbooks/k8s-cluster.yml unconditionally runs
git submodule update --init from the repository root, so an extracted
tree without Git metadata fails before Kubespray starts. With submodules
already initialized at the pinned commits, that task is an offline no-op.
Run this from the checkout's parent directory after setup.sh has
populated its Galaxy dependencies and
git submodule update --init --recursive has completed:
git -C deepops submodule status --recursive # every line must start with a space (initialized, pinned)
tar --exclude='deepops/config' \
-czf /tmp/deepops-source.tar.gz deepops
The archive is larger with Git metadata included; that is the price of the
Kubernetes path working. Excluding deepops/config still keeps site
secrets out of the transfer artifact.
Package repositories and images for approved removable-media transfer and
create checksums. Substitute a protected, user-owned staging directory for
/path/to/staging; create the ISO files without elevated privileges:
Transfer the ISO files, checkout archive, and checksum file through the site's
approved boundary process. Verify the checksums again
inside the isolated network before importing anything:
cd /path/to/import
sha256sum -c SHA256SUMS
4. Import artifacts on the isolated side
Mount the verified transfer images and copy their contents to persistent or
working directories:
sudomkdir -p /mnt/deepops-packages /mnt/deepops-images \
/mnt/deepops-charts /mnt/deepops-kubespray /mnt/deepops-keys
sudo mount -o loop /path/to/import/packages.iso /mnt/deepops-packages
sudo mount -o loop /path/to/import/images.iso /mnt/deepops-images
sudo mount -o loop /path/to/import/charts.iso /mnt/deepops-charts
sudo mount -o loop /path/to/import/kubespray-offline.iso /mnt/deepops-kubespray
sudo mount -o loop /path/to/import/keys.iso /mnt/deepops-keys
sudomkdir -p /var/repos /var/www/html/charts
mkdir -p /tmp/images
sudocp -a /mnt/deepops-packages/. /var/repos/
cp -a /mnt/deepops-images/. /tmp/images/
sudocp -a /mnt/deepops-charts/. /var/www/html/charts/
mkdir -p /tmp/kubespray-offline
cp -a /mnt/deepops-kubespray/. /tmp/kubespray-offline/
Publish signing keys first, and fail if any expected file is missing. This
EL9/Ubuntu 24.04 example corresponds exactly to the connected-side commands:
For APT, preserve apt-mirror's hostname and path layout verbatim:
sudo install -d /var/www/html/repos
test -d /var/repos/mirror/archive.ubuntu.com/ubuntu/dists/noble
test -d /var/repos/mirror/download.docker.com/linux/ubuntu/dists/noble
test -d /var/repos/mirror/nvidia.github.io/libnvidia-container/stable/deb/amd64
test -d /var/repos/mirror/developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64
sudocp -a /var/repos/mirror/. /var/www/html/repos/
Publish the direct Slurm/Enroot files under the paths used by the generated
overrides:
test -d /var/repos/downloads
sudo install -d /var/www/html/downloads
sudocp -a /var/repos/downloads/. /var/www/html/downloads/
test -z "$(find /var/www/html/downloads -maxdepth 1 -type f -size 0 -print -quit)"
For RPM, publish the exact top-level repo-ID directories, EPEL bootstrap
RPM, and a rewritten Container Toolkit repo file whose base URLs are
internal:
Replace package-server with the real internal DNS name in both the repo
file and step 5. Verify every configured file URL with curl -fsS and every
APT/RPM metadata path with the target package manager before deployment.
Preserve GPG verification.
Load the registry image and start the isolated registry if the site does
not already provide one. Run this fallback on the registry host itself. It
is plain HTTP, so the imports below push through Docker's IPv4 loopback
exception (127.0.0.1:5000); cluster nodes use the reachable
registry-host:5000 name configured later. Do not push to
registry-host:5000 from the importer unless that Docker daemon has been
separately and deliberately configured for the insecure endpoint:
Import, retag, and push every archived image on that registry host. Preserve
the complete repository path expected by the consuming chart or role. The
hostname used for the push is not part of the stored repository path, so a
cluster pull of registry-host:5000/nvidia/cuda:... resolves the image
pushed as 127.0.0.1:5000/nvidia/cuda:...:
Extract the initialized DeepOps checkout on the provisioning machine:
mkdir -p "$HOME/deepops-src"
tar -xzf /path/to/import/deepops-source.tar.gz -C "$HOME/deepops-src"cd"$HOME/deepops-src/deepops"
Activate the provisioning environment prepared before isolation (or install
it from approved internal indexes), then copy and edit config.example/ if
site configuration was not transferred separately. Install the mirrored
Helm version expected by scripts/k8s/install_helm.sh on an amd64 Linux
provisioning host so that script does not fetch its public installer:
Mirror the matching archive for another provisioning platform; do not use
the amd64 archive there.
Publish the Kubespray static-file tree, then import every Kubespray, GPU
Operator, and NFS image collected in step 3 into the registry. Do not run
the pinned manager's register action: before importing, it overwrites
/etc/docker/daemon.json or /etc/containers/registries.conf, substitutes
the importer's hostname instead of DESTINATION_REGISTRY, and does not back
up, restore, or restart the runtime. Extract its archive and perform the same
load/tag/push loop without changing host configuration:
sudo install -d /var/www/html/kubespray
sudo tar -xzf /tmp/kubespray-offline/offline-files.tar.gz \
-C /var/www/html/kubespray
test -d /var/www/html/kubespray/offline-files
task_kubespray_import=$(mktemp -d)
(
set -euo pipefail
trap'rm -rf "${task_kubespray_import}"' EXIT
tar -xzf /tmp/kubespray-offline/container-images.tar.gz \
-C "${task_kubespray_import}"whileread -r archive repository; dotest -n "${archive}" && test -n "${repository}"
load_line=$(docker load \
-i "${task_kubespray_import}/container-images/${archive}" | sed -n '1p')
source_ref=$(printf'%s\n'"${load_line}" | awk '{print $3}')
if [ "${source_ref}" = ID: ]; then
source_ref=$(printf'%s\n'"${load_line}" | awk '{print $4}')
image_id=$(docker image inspect --format )
-n
docker tag
docker push
<
)
5. Configure DeepOps for internal endpoints
Set mirror overrides in config/group_vars/all.yml (or a narrower group file).
Use values matching the paths actually published by the site:
docker_ubuntu_repo_base_url:"http://package-server/repos/download.docker.com/linux/ubuntu"docker_ubuntu_repo_gpgkey:"http://package-server/keys/docker-ubuntu.asc"nvidia_container_toolkit_repo_base_url:"http://package-server/repos/nvidia.github.io/libnvidia-container"nvidia_container_toolkit_repo_gpg_url:"http://package-server/keys/libnvidia-container.asc"docker_rh_repo_base_url:"http://package-server/repos/docker-ce-stable"docker_rh_repo_gpgkey:"http://package-server/keys/docker-rpm.asc"nvidia_container_toolkit_rpm_repo_url:"http://package-server/repos/nvidia-container-toolkit.repo"docker_insecure_registries:-"registry-host:5000"# CUDA repository mirror (required by the default Slurm path; see below)nvidia_driver_ubuntu_cuda_keyring_url:"http://package-server/repos/developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb"nvidia_driver_rhel_cuda_repo_baseurl:"http://package-server/repos/cuda-rhel9-x86_64/"nvidia_driver_rhel_cuda_repo_gpgkey:"http://package-server/keys/cuda-rhel9-D42D0685.pub"# EPEL mirror (Enterprise Linux; key import fails offline without this)epel_package:"http://package-server/repos/epel-release-latest-9.noarch.rpm"epel_key_url:"http://package-server/keys/RPM-GPG-KEY-EPEL-9"
For Ubuntu 24.04, prepare every target before running the Slurm playbook or
any play that includes nvidia_cuda or nvidia_dcgm. Replace
package-server with the real internal host first. These commands install the
mirrored keyring, remove the public source that package creates, write the
internal cuda-compute-repo.list, update that source in isolation, and then
prove the complete APT update resolves no public NVIDIA host before allowing a
normal update:
Use the matching repository directory and keyring-generated source filename on
another Ubuntu release. Keep all other APT sources internal too: the final
apt-get update intentionally checks the complete target configuration, which
is what the role will update. Retain the
nvidia_driver_ubuntu_cuda_keyring_url override so the role finds the mirrored
package; do not set nvidia_driver_ubuntu_cuda_repo_baseurl expecting it to
redirect APT.
For Slurm, disable the default pull-through registry cache when upstream
Docker Hub is unreachable. If using the site's registry, the smallest honest
profile is:
Mirror and override the Slurm source/archive variables listed above. If their
artifacts were deliberately excluded, disable the corresponding optional
features, for example slurm_enable_monitoring, slurm_install_hpcsdk,
slurm_install_nhc, slurm_install_enroot, or slurm_install_pyxis. Do not
disable a required feature merely to make a playbook pass.
For Kubernetes, add the pinned Kubespray offline rewrites and the DeepOps chart
repositories. This layout matches the static-file extraction and image-manager
registration commands above:
Use trusted TLS and skip_verify: false when the internal registry provides
it. Set k8s_nfs_client_provisioner: false only if the cluster uses a
site-owned storage path or intentionally has no dynamic NFS provisioner.
The current top-level Kubernetes playbook invokes a Helm installer URL and
adds https://charts.helm.sh/stable; when the Ansible host platform differs
from the cluster nodes (for example a macOS or arm64 control machine), its
artifact step also downloads kubectl and its checksum directly from
https://dl.k8s.io. Run the deployment from a control host matching the
cluster platform so the playbook fetches kubectl from a cluster node
instead, or pre-approve an internal mirror for that URL. Its containerd
local-registry settings
also carry an explicit TODO in config.example/group_vars/k8s_cluster.yml.
Therefore, before declaring a fully disconnected Kubernetes run ready, prove
that these references are satisfied by approved internal endpoints or obtain a
DeepOps code change. Do not claim that the current playbook is turnkey offline.
6. Preflight and deploy
Run the machine-readable doctor from the isolated provisioning machine:
Require exit code 0 and top-level "ok": true. Review the reported host
count/groups against the approved inventory. The GPU PCI count is informational;
all other failed checks must be understood and resolved before deployment.
Require exit code 0, "ok": true, nodes_ready == nodes_total,
"gpus_allocatable" greater than zero, "cuda_smoke_ok": true, and
"cuda_smoke_gpus" greater than zero. A play recap with failed=0 is not
the success gate.
Cleanup
On Kubernetes smoke success, the validator deletes the temporary
deepops-validate namespace automatically. On failure it deliberately keeps
the namespace for diagnosis; delete it only after collecting evidence:
Unmount/eject approved transfer media and remove temporary extracted copies
only after checksums, repository reachability, image pulls, and the required
validation record have been captured:
Do not remove the internal mirrors or registry: deployed nodes continue to
need them for repairs and rescheduling.
Remove secrets and site inventory from transfer staging. Retain the manifest,
checksums, exact DeepOps commit, mirror snapshot/version, play recap, and
validator JSON according to site policy.
Delete /tmp/kubespray-effective-vars.json after artifact collection. It is a
resolved inventory export and may contain site variables that do not belong in
the transfer set or retained validation evidence.
Source-derived failure branches
These branches follow from checked-in tasks, defaults, and validators; they are
not claims that this complete procedure was live-tested in an air gap.
build_offline_cache.sh fails because
playbooks/airgap/build-offline-cache.yml is missing: this automation was
removed as broken/unsupported. Use the explicit mirror workflow above.
Doctor reports kubespray_submodule_initialized: false, or Kubernetes
fails on kubespray_defaults imports: transfer an archive made after
git submodule update --init --recursive; do not fetch from the isolated
side.
Generated Kubernetes lists contain a default version instead of the
configured override: confirm that the representative host resolves the
override, regenerate its effective-variable JSON, and pass it with -e @....
The generator's localhost play does not inherit k8s_cluster variables from
-i alone.
An image push reports an HTTPS response error against the fallback HTTP
registry: perform the import on the registry host through
127.0.0.1:5000. For a remote registry, use trusted TLS or stop until the
importer's runtime is deliberately configured and restarted for that exact
endpoint; the target-node docker_insecure_registries value does not change
the importer.
Package task tries a public URL or reports a missing package: the mirror
set or variable override is incomplete for the enabled profile. Add the exact
repository/file to the connected-side manifest, transfer a new signed
snapshot, and rerun the same idempotent playbook.
Slurm play reaches GitHub, SchedMD, Open MPI, or NVIDIA download hosts:
mirror and override the applicable source URL, or intentionally disable the
optional NHC, Pyxis, HPC SDK, or monitoring component.
Bare nvidia-smi over SSH reports No devices were found on a Slurm
node: DeepOps may hide GPUs from ordinary SSH sessions. Trust the
validator's srun job; do not diagnose the driver from the bare SSH result.
Slurm validator reports gpu_job_ok: false: inspect its failures list.
If it reports driver errors, follow skills/diagnose-driver-install/; if the
node is down/drained, inspect its Slurm reason before resuming it.
Kubernetes play tries raw.githubusercontent.com,
charts.helm.sh/stable, a public chart repo, or a public image registry:
stop. The pinned Helm/chart/image dependency is not yet served internally;
complete that mirror or gate the required DeepOps change before rerunning.
Kubernetes CUDA smoke is Pending or shows an image pull error: confirm
the --cuda-image path exists in the internal registry and is reachable by
every GPU node. Inspect the retained deepops-validate namespace before
cleanup.
Kubernetes reports gpus_allocatable: 0: wait for the mirrored GPU stack
to become ready, then diagnose GPU Operator/device-plugin or driver failures.
Do not use --allow-no-gpus as a deployment success gate for a GPU cluster.
A rerun uses facts from a reprovisioned/reused hostname: add
--flush-cache to the same playbook command.
The PRIVATE_REGISTRY=nvcr.io setting makes the checked-in image manager
strip nvcr.io/ just as it strips the Kubespray source registries, so an
internal registry mirror can serve the paths that the GPU Operator chart
requests. If NFS is disabled, omit its helm template command. If site
overrides change any GPU Operator flag, render with those values instead.
The inventory path is still required by the generator, but the explicit
effective-variable JSON is what makes group/host version overrides visible
to its localhost play. Treat a missing representative host, an empty list, a
failed pull, or an unparseable rendered image as a collection failure; do not
continue with a partial archive.
These commands assume the fallback plain-HTTP registry is local to the
importer. For a site registry on another host, require trusted TLS or have
the site administrator configure and restart the importer's runtime under
site change control before substituting that registry endpoint. Confirm the
imported repositories through the registry's normal catalog or pull checks
before deployment.
