deploy-airgapped

name	deploy-airgapped
description	Prepare mirrors and transfer artifacts, configure DeepOps, deploy Slurm or Kubernetes GPU clusters without Internet access, and validate them with machine-readable gates. Use for disconnected, restricted-egress, offline, or air-gapped DeepOps installations and for diagnosing missing package, file, chart, or container artifacts.

Deploy DeepOps in an air-gapped environment

Preconditions and boundaries

• Use an Internet-connected staging machine and a separate provisioning machine that can reach every target and every internal mirror. Do not connect an isolated network to the Internet for this procedure.
• Fix one DeepOps commit, target OS release, CPU architecture, cluster type, inventory, and enabled feature set before mirroring. Repository and image requirements change when any of these inputs change.
• Provide complete distribution package mirrors plus internal HTTP package/file service and container registry endpoints. Use trusted TLS where possible; if the registry is intentionally insecure, configure it explicitly.
• Bootstrap the isolated provisioning machine with the OS packages and Python environment required by scripts/setup.sh before closing the boundary, or provide approved internal OS and Python package indexes. Copying the source tree alone does not install Ansible.
• Reserve enough storage for repository metadata, packages, container images, charts, the initialized DeepOps checkout, and two copies during transfer.
• Confirm that every target is fully manageable and has no active users or workloads. Driver installation can reboot hosts.
• Run DeepOps commands from the repository root. Keep secrets out of archives and store site configuration in config/, never config.example/.

DeepOps does not provide supported mirror-building automation. Do not run scripts/airgap/build_offline_cache.sh: it calls the removed playbooks/airgap/build-offline-cache.yml and cannot complete in this tree. Build and verify the mirrors explicitly as described below.

1. Freeze the deployment inputs

1. Record the DeepOps commit and initialize every submodule on the connected staging machine:

   git rev-parse HEAD
   git submodule update --init --recursive
   ./scripts/setup.sh
   

2. Copy config.example/ to config/. Define the real hosts in config/inventory before collecting artifacts so the selected cluster path and optional components are known:

   • Slurm: controller/login hosts in [slurm-master], compute hosts in [slurm-node]; a single host may be in both groups.
   • Kubernetes: control-plane hosts in [kube_control_plane] and [etcd], workers in [kube_node]; a single host may be in all three groups.

3. Review the resolved inventory. Stop if it contains any unintended host:

   ansible-inventory -i config/inventory --list
   

4. Record the target OS release/architecture and all enabled roles. Mirror every dependency of that exact profile. In particular, account for:

   • distribution, Docker CE, NVIDIA CUDA/driver, NVIDIA Container Toolkit, and EPEL repositories where applicable;
   • direct-download archives referenced by enabled role defaults;
   • every container image and Helm chart used by the chosen Slurm or Kubernetes path;
   • the initialized Kubespray submodule, Galaxy roles/collections, and Python packages needed on the provisioning machine.

2. Build package mirrors on the connected side

Ubuntu/APT

Install apt-mirror, set base_path in /etc/apt/mirror.list (for example, /var/repos), and add the suites for the target release. For Ubuntu 24.04 (noble), the current DeepOps documentation uses these sources:

deb http://archive.ubuntu.com/ubuntu noble main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu noble-security main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu noble-updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu noble-backports main restricted universe multiverse
deb https://download.docker.com/linux/ubuntu noble stable
deb https://nvidia.github.io/libnvidia-container/stable/deb/amd64 /
deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64 /

Add DGX OS or MAAS repositories only when the selected profile uses them. Use the matching suite and architecture instead of noble/amd64 for another target. Download exactly the configured set:

sudo mkdir -p /var/repos
sudo apt-mirror

Download the signing material that DeepOps fetches directly, plus the RPM bootstrap files when building an Enterprise Linux mirror. Keep these outside mirror/ so their archive paths are deterministic:

sudo mkdir -p /var/repos/keys /var/repos/bootstrap
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  -o /var/repos/keys/docker-ubuntu.asc
sudo curl -fsSL https://download.docker.com/linux/centos/gpg \
  -o /var/repos/keys/docker-rpm.asc
sudo curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  -o /var/repos/keys/libnvidia-container.asc
sudo curl -fsSL \
  https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/3bf863cc.pub \
  -o /var/repos/keys/cuda-ubuntu2404-3bf863cc.pub
sudo curl -fsSL \
  https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/D42D0685.pub \
  -o /var/repos/keys/cuda-rhel9-D42D0685.pub
sudo curl -fsSL \
  https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-9 \
  -o /var/repos/keys/RPM-GPG-KEY-EPEL-9
sudo curl -fsSL \
  https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm \
  -o /var/repos/bootstrap/epel-release-latest-9.noarch.rpm
gpg --show-keys --with-fingerprint /var/repos/keys/*

Replace ubuntu2404, rhel9, and 9 together for another supported target. Verify the downloaded fingerprints against the vendor's published values under the site's package-trust process before transfer; do not turn off GPG checking.

Enterprise Linux/RPM

Configure the target distribution repositories plus EPEL, Docker CE, CUDA, libnvidia-container, and nvidia-container-toolkit. For EL9, download the checked-in documentation's complete additional-repository set into the top-level repo-ID directories created by reposync:

for repo_id in epel docker-ce-stable cuda-rhel9-x86_64 \
  libnvidia-container nvidia-container-toolkit; do
  sudo reposync -l --repoid="${repo_id}" --downloadcomps \
    --download-metadata --download_path=/var/repos
  sudo createrepo "/var/repos/${repo_id}"
done

Use cuda-rhel8-x86_64 instead on EL8. Compare the final IDs and package contents with /etc/yum.repos.d/ and the enabled workload on a connected reference host; distribution repositories remain a separate prerequisite.

3. Mirror direct files, charts, and images

1. Download the direct files selected by the current default Slurm profile. These names and versions come from the checked-in role defaults:

   sudo install -d -m 0755 /var/repos/downloads
   while read -r url name; do
     sudo curl -fL "${url}" -o "/var/repos/downloads/${name}"
   done <<'EOF'
   https://download.schedmd.com/slurm/slurm-26.05.1.tar.bz2 slurm-26.05.1.tar.bz2
   https://download.open-mpi.org/release/hwloc/v2.5/hwloc-2.5.0.tar.gz hwloc-2.5.0.tar.gz
   https://github.com/openpmix/openpmix/releases/download/v3.2.3/pmix-3.2.3.tar.bz2 pmix-3.2.3.tar.bz2
   https://github.com/mej/nhc/releases/download/1.4.3/lbnl-nhc-1.4.3.tar.xz lbnl-nhc-1.4.3.tar.xz
   https://github.com/NVIDIA/pyxis/archive/v0.11.1.tar.gz pyxis-0.11.1.tar.gz
   https://developer.download.nvidia.com/hpc-sdk/23.7/nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz
   https://get.helm.sh/helm-v3.17.1-linux-amd64.tar.gz helm-v3.17.1-linux-amd64.tar.gz
   EOF
   for name in slurm-26.05.1.tar.bz2 hwloc-2.5.0.tar.gz \
     pmix-3.2.3.tar.bz2 lbnl-nhc-1.4.3.tar.xz pyxis-0.11.1.tar.gz \
     nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz \
     helm-v3.17.1-linux-amd64.tar.gz; do
     test -s "/var/repos/downloads/${name}"
   done
   

   Enroot is also enabled by default, but its package filenames are owned by the pinned nvidia.enroot Galaxy role rather than this repository. After ./scripts/setup.sh, use that installed role's real package list to download the target-OS packages and generate the matching Ansible override. Choose enroot_deb_packages for Ubuntu or enroot_rpm_packages for Enterprise Linux; repeat for both when the inventory mixes OS families:

   test -s roles/galaxy/nvidia.enroot/defaults/main.yml
   cat >/tmp/deepops-mirror-enroot.yml <<'EOF'
   ---
   - hosts: localhost
     connection: local
     gather_facts: true
     become: 
     vars_files:
       - 
     tasks:
       - name: Download the pinned Enroot packages
         get_url:
           url: 
           dest: 
           mode: 
         loop: 
       - name: Write the matching offline override
         copy:
           dest: 
           mode: 
           content: |
             {{ enroot_package_list_var }}:
             {%  package_url  lookup(, enroot_package_list_var) %}
               - 
             {% endfor %}
   EOF
   ansible-playbook -i localhost, /tmp/deepops-mirror-enroot.yml \
     -e  \
     -e enroot_package_list_var=enroot_deb_packages \
     -e offline_http_base=http://package-server/downloads
   

Transfer the ISO files, checkout archive, and checksum file through the site's approved boundary process. Verify the checksums again inside the isolated network before importing anything:

cd /path/to/import
sha256sum -c SHA256SUMS

4. Import artifacts on the isolated side

1. Mount the verified transfer images and copy their contents to persistent or working directories:

   sudo mkdir -p /mnt/deepops-packages /mnt/deepops-images \
     /mnt/deepops-charts /mnt/deepops-kubespray /mnt/deepops-keys
   sudo mount -o loop /path/to/import/packages.iso /mnt/deepops-packages
   sudo mount -o loop /path/to/import/images.iso /mnt/deepops-images
   sudo mount -o loop /path/to/import/charts.iso /mnt/deepops-charts
   sudo mount -o loop /path/to/import/kubespray-offline.iso /mnt/deepops-kubespray
   sudo mount -o loop /path/to/import/keys.iso /mnt/deepops-keys
   sudo mkdir -p /var/repos /var/www/html/charts
   mkdir -p /tmp/images
   sudo cp -a /mnt/deepops-packages/. /var/repos/
   cp -a /mnt/deepops-images/. /tmp/images/
   sudo cp -a /mnt/deepops-charts/. /var/www/html/charts/
   mkdir -p /tmp/kubespray-offline
   cp -a /mnt/deepops-kubespray/. /tmp/kubespray-offline/
   

2. Publish signing keys first, and fail if any expected file is missing. This EL9/Ubuntu 24.04 example corresponds exactly to the connected-side commands:

   sudo install -d /var/www/html/keys
   for key in docker-ubuntu.asc docker-rpm.asc libnvidia-container.asc \
     cuda-ubuntu2404-3bf863cc.pub cuda-rhel9-D42D0685.pub \
     RPM-GPG-KEY-EPEL-9; do
     test -s "/mnt/deepops-keys/${key}"
     sudo install -m 0644 "/mnt/deepops-keys/${key}" "/var/www/html/keys/${key}"
   done
   gpg --show-keys --with-fingerprint /var/www/html/keys/*
   sha256sum /var/www/html/keys/*
   

   For APT, preserve apt-mirror's hostname and path layout verbatim:

5. Configure DeepOps for internal endpoints

Set mirror overrides in config/group_vars/all.yml (or a narrower group file). Use values matching the paths actually published by the site:

docker_ubuntu_repo_base_url: "http://package-server/repos/download.docker.com/linux/ubuntu"
docker_ubuntu_repo_gpgkey: "http://package-server/keys/docker-ubuntu.asc"
nvidia_container_toolkit_repo_base_url: "http://package-server/repos/nvidia.github.io/libnvidia-container"
nvidia_container_toolkit_repo_gpg_url: "http://package-server/keys/libnvidia-container.asc"

docker_rh_repo_base_url: "http://package-server/repos/docker-ce-stable"
docker_rh_repo_gpgkey: "http://package-server/keys/docker-rpm.asc"
nvidia_container_toolkit_rpm_repo_url: "http://package-server/repos/nvidia-container-toolkit.repo"

docker_insecure_registries:
  - "registry-host:5000"

# CUDA repository mirror (required by the default Slurm path; see below)
nvidia_driver_ubuntu_cuda_keyring_url: "http://package-server/repos/developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb"
nvidia_driver_rhel_cuda_repo_baseurl: "http://package-server/repos/cuda-rhel9-x86_64/"
nvidia_driver_rhel_cuda_repo_gpgkey: "http://package-server/keys/cuda-rhel9-D42D0685.pub"

# EPEL mirror (Enterprise Linux; key import fails offline without this)
epel_package: "http://package-server/repos/epel-release-latest-9.noarch.rpm"
epel_key_url: "http://package-server/keys/RPM-GPG-KEY-EPEL-9"

For Ubuntu 24.04, prepare every target before running the Slurm playbook or any play that includes nvidia_cuda or nvidia_dcgm. Replace package-server with the real internal host first. These commands install the mirrored keyring, remove the public source that package creates, write the internal cuda-compute-repo.list, update that source in isolation, and then prove the complete APT update resolves no public NVIDIA host before allowing a normal update:

(
  set -euo pipefail
  task_cuda_repo_url='http://package-server/repos/developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64'
  task_cuda_keyring_url="${task_cuda_repo_url}/cuda-keyring_1.1-1_all.deb"

  case "${task_cuda_repo_url}" in
    *package-server*|https://developer.download.nvidia.com/*|http://developer.download.nvidia.com/*)
      echo 'ERROR: set task_cuda_repo_url to the internal CUDA mirror' >&2
      exit 1
      ;;
  esac

  task_cuda_keyring_deb=$(mktemp)
  task_apt_uris=$(mktemp)
  trap 'rm -f "${task_cuda_keyring_deb}" "${task_apt_uris}"' EXIT
  curl -fsSL "${task_cuda_keyring_url}" -o "${task_cuda_keyring_deb}"
  test -s "${task_cuda_keyring_deb}"
  sudo dpkg -i "${task_cuda_keyring_deb}"

  if [ -e /etc/apt/sources.list.d/cuda-ubuntu2404-x86_64.list ]; then
    sudo mv --backup=numbered -- \
      /etc/apt/sources.list.d/cuda-ubuntu2404-x86_64.list \
      /etc/apt/sources.list.d/cuda-ubuntu2404-x86_64.list.disabled
  fi
  printf 'deb [signed-by=/usr/share/keyrings/cuda-archive-keyring.gpg] %s /\n' \
    "${task_cuda_repo_url%/}" |
    sudo tee /etc/apt/sources.list.d/cuda-compute-repo.list >/dev/null

  sudo apt-get \
    -o Dir::Etc::sourcelist='sources.list.d/cuda-compute-repo.list' \
    -o Dir::Etc::sourceparts='-' update

  apt-get --print-uris update |  
  task_public_nvidia_urls=$(
    sed -n   |
      awk -F/ 
  )
   [ -n  ]; 
      \
       >&2
     1
  
   apt-get update
)

Use the matching repository directory and keyring-generated source filename on another Ubuntu release. Keep all other APT sources internal too: the final apt-get update intentionally checks the complete target configuration, which is what the role will update. Retain the nvidia_driver_ubuntu_cuda_keyring_url override so the role finds the mirrored package; do not set nvidia_driver_ubuntu_cuda_repo_baseurl expecting it to redirect APT.

For Slurm, disable the default pull-through registry cache when upstream Docker Hub is unreachable. If using the site's registry, the smallest honest profile is:

slurm_enable_container_registry: false
standalone_container_registry_cache_enable: false
slurm_exporter_build_image: false
slurm_exporter_container: "registry-host:5000/deepops/prometheus-slurm-exporter:2.0.0"
prometheus_container: "registry-host:5000/prom/prometheus:v3.13.0"
grafana_container: "registry-host:5000/grafana/grafana:13.1.0"
alertmanager_container: "registry-host:5000/prom/alertmanager:v0.33.0"
node_exporter_container: "registry-host:5000/prometheus/node-exporter:v1.11.1"
nvidia_dcgm_container: "registry-host:5000/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless"

Mirror and override the Slurm source/archive variables listed above. If their artifacts were deliberately excluded, disable the corresponding optional features, for example slurm_enable_monitoring, slurm_install_hpcsdk, slurm_install_nhc, slurm_install_enroot, or slurm_install_pyxis. Do not disable a required feature merely to make a playbook pass.

For Kubernetes, add the pinned Kubespray offline rewrites and the DeepOps chart repositories. This layout matches the static-file extraction and image-manager registration commands above:

registry_host: "registry-host:5000"
registry_addr: "registry-host:5000"
files_repo: "http://package-server/kubespray/offline-files"

kube_image_repo: "{{ registry_host }}"
gcr_image_repo: "{{ registry_host }}"
docker_image_repo: "{{ registry_host }}"
quay_image_repo: "{{ registry_host }}"
github_image_repo: "{{ registry_host }}"
github_url: "{{ files_repo }}/github.com"
dl_k8s_io_url: "{{ files_repo }}/dl.k8s.io"
storage_googleapis_url: "{{ files_repo }}/storage.googleapis.com"
get_helm_url: "{{ files_repo }}/get.helm.sh"
local_path_provisioner_helper_image_repo: "{{ registry_host }}/busybox"

gpu_operator_helm_repo: "http://package-server/charts"
k8s_nfs_client_helm_repo: "http://package-server/charts"

containerd_registries_mirrors:
  - prefix: "{{ registry_addr }}"
    mirrors:
      - host: "http://"
         [, ]
         
    
    
        
         [, ]
         
    
    
        
         [, ]
         
    
    
        
         [, ]
         
    
    
        
         [, ]
         
    
    
        
         [, ]
         
    
    
        
         [, ]
         

Use trusted TLS and skip_verify: false when the internal registry provides it. Set k8s_nfs_client_provisioner: false only if the cluster uses a site-owned storage path or intentionally has no dynamic NFS provisioner.

The current top-level Kubernetes playbook invokes a Helm installer URL and adds https://charts.helm.sh/stable; when the Ansible host platform differs from the cluster nodes (for example a macOS or arm64 control machine), its artifact step also downloads kubectl and its checksum directly from https://dl.k8s.io. Run the deployment from a control host matching the cluster platform so the playbook fetches kubectl from a cluster node instead, or pre-approve an internal mirror for that URL. Its containerd local-registry settings also carry an explicit TODO in config.example/group_vars/k8s_cluster.yml. Therefore, before declaring a fully disconnected Kubernetes run ready, prove that these references are satisfied by approved internal endpoints or obtain a DeepOps code change. Do not claim that the current playbook is turnkey offline.

6. Preflight and deploy

Run the machine-readable doctor from the isolated provisioning machine:

python3 scripts/validation/deepops_doctor.py --remote --json

Require exit code 0 and top-level "ok": true. Review the reported host count/groups against the approved inventory. The GPU PCI count is informational; all other failed checks must be understood and resolved before deployment.

Slurm path

ansible-playbook -l slurm-cluster playbooks/slurm-cluster.yml

Rerun the same command after a transient mirror/package error. Validate on a Slurm controller, login, or compute node:

python3 scripts/validation/validate_slurm.py --json

Require exit code 0, "ok": true, "nodes_unavailable": 0, "gpus_configured" greater than zero, and "gpu_job_ok": true.

Kubernetes path

Run only after all Kubernetes package, file, chart, and image references have been resolved to the isolated environment:

ansible-playbook -l k8s_cluster playbooks/k8s-cluster.yml

Validate from a machine whose kubectl context reaches the cluster, overriding the public default CUDA image with the imported registry image:

python3 scripts/validation/validate_k8s.py --json --cuda-smoke \
  --cuda-image registry-host:5000/nvidia/cuda:12.4.1-base-ubuntu22.04

Require exit code 0, "ok": true, nodes_ready == nodes_total, "gpus_allocatable" greater than zero, "cuda_smoke_ok": true, and "cuda_smoke_gpus" greater than zero. A play recap with failed=0 is not the success gate.

Cleanup

• On Kubernetes smoke success, the validator deletes the temporary deepops-validate namespace automatically. On failure it deliberately keeps the namespace for diagnosis; delete it only after collecting evidence:

  kubectl delete namespace deepops-validate --ignore-not-found
  

• Unmount/eject approved transfer media and remove temporary extracted copies only after checksums, repository reachability, image pulls, and the required validation record have been captured:

  sudo umount /mnt/deepops-packages
  sudo umount /mnt/deepops-images
  sudo umount /mnt/deepops-charts
  sudo umount /mnt/deepops-kubespray
  sudo umount /mnt/deepops-keys
  

  Do not remove the internal mirrors or registry: deployed nodes continue to need them for repairs and rescheduling.

• Remove secrets and site inventory from transfer staging. Retain the manifest, checksums, exact DeepOps commit, mirror snapshot/version, play recap, and validator JSON according to site policy.

• Delete /tmp/kubespray-effective-vars.json after artifact collection. It is a resolved inventory export and may contain site variables that do not belong in the transfer set or retained validation evidence.

Source-derived failure branches

These branches follow from checked-in tasks, defaults, and validators; they are not claims that this complete procedure was live-tested in an air gap.

• build_offline_cache.sh fails because playbooks/airgap/build-offline-cache.yml is missing: this automation was removed as broken/unsupported. Use the explicit mirror workflow above.
• Doctor reports kubespray_submodule_initialized: false, or Kubernetes fails on kubespray_defaults imports: transfer an archive made after git submodule update --init --recursive; do not fetch from the isolated side.
• Generated Kubernetes lists contain a default version instead of the configured override: confirm that the representative host resolves the override, regenerate its effective-variable JSON, and pass it with -e @.... The generator's localhost play does not inherit k8s_cluster variables from -i alone.
• An image push reports an HTTPS response error against the fallback HTTP registry: perform the import on the registry host through 127.0.0.1:5000. For a remote registry, use trusted TLS or stop until the importer's runtime is deliberately configured and restarted for that exact endpoint; the target-node docker_insecure_registries value does not change the importer.
• Package task tries a public URL or reports a missing package: the mirror set or variable override is incomplete for the enabled profile. Add the exact repository/file to the connected-side manifest, transfer a new signed snapshot, and rerun the same idempotent playbook.
• Slurm play reaches GitHub, SchedMD, Open MPI, or NVIDIA download hosts: mirror and override the applicable source URL, or intentionally disable the optional NHC, Pyxis, HPC SDK, or monitoring component.
• Bare nvidia-smi over SSH reports No devices were found on a Slurm node: DeepOps may hide GPUs from ordinary SSH sessions. Trust the validator's srun job; do not diagnose the driver from the bare SSH result.
• Slurm validator reports gpu_job_ok: false: inspect its failures list. If it reports driver errors, follow skills/diagnose-driver-install/; if the node is down/drained, inspect its Slurm reason before resuming it.
• Kubernetes play tries raw.githubusercontent.com, charts.helm.sh/stable, a public chart repo, or a public image registry: stop. The pinned Helm/chart/image dependency is not yet served internally; complete that mirror or gate the required DeepOps change before rerunning.