deploy-airgapped

# Deploy DeepOps in an air-gapped environment ## Preconditions and boundaries - Use an Internet-connected staging machine and a separate provisioning machine that can reach every target and every internal mirror. Do not connect an isolated network to the Internet for this procedure. - Fix one DeepOps commit, target OS release, CPU architecture, cluster type, inventory, and enabled feature set before mirroring. Repository and image requirements change when any of these inputs change. - Provide complete distribution package mirrors plus internal HTTP package/file service and container registry endpoints. Use trusted TLS where possible; if the registry is intentionally insecure, configure it explicitly. - Bootstrap the isolated provisioning machine with the OS packages and Python environment required by `scripts/setup.sh` before closing the boundary, or provide approved internal OS and Python package indexes. Copying the source tree alone does not install Ansible. - Reserve enough storage for repository metadata, packages, container images, charts, the initialized DeepOps checkout, and two copies during transfer. - Confirm that every target is fully manageable and has no active users or workloads. Driver installation can reboot hosts. - Run DeepOps commands from the repository root. Keep secrets out of archives and store site configuration in `config/`, never `config.example/`. DeepOps does not provide supported mirror-building automation. Do **not** run `scripts/airgap/build_offline_cache.sh`: it calls the removed `playbooks/airgap/build-offline-cache.yml` and cannot complete in this tree. Build and verify the mirrors explicitly as described below. ## 1. Freeze the deployment inputs 1. Record the DeepOps commit and initialize every submodule on the connected staging machine: ```bash git rev-parse HEAD git submodule update --init --recursive ./scripts/setup.sh ``` 2. Copy `config.example/` to `config/`. Define the real hosts in `config/inventory` before collecting artifacts so the selected cluster path and optional components are known: - Slurm: controller/login hosts in `[slurm-master]`, compute hosts in `[slurm-node]`; a single host may be in both groups. - Kubernetes: control-plane hosts in `[kube_control_plane]` and `[etcd]`, workers in `[kube_node]`; a single host may be in all three groups. 3. Review the resolved inventory. Stop if it contains any unintended host: ```bash ansible-inventory -i config/inventory --list ``` 4. Record the target OS release/architecture and all enabled roles. Mirror every dependency of that exact profile. In particular, account for: - distribution, Docker CE, NVIDIA CUDA/driver, NVIDIA Container Toolkit, and EPEL repositories where applicable; - direct-download archives referenced by enabled role defaults; - every container image and Helm chart used by the chosen Slurm or Kubernetes path; - the initialized Kubespray submodule, Galaxy roles/collections, and Python packages needed on the provisioning machine. ## 2. Build package mirrors on the connected side ### Ubuntu/APT Install `apt-mirror`, set `base_path` in `/etc/apt/mirror.list` (for example, `/var/repos`), and add the suites for the target release. For Ubuntu 24.04 (`noble`), the current DeepOps documentation uses these sources: ```text deb http://archive.ubuntu.com/ubuntu noble main restricted universe multiverse deb http://archive.ubuntu.com/ubuntu noble-security main restricted universe multiverse deb http://archive.ubuntu.com/ubuntu noble-updates main restricted universe multiverse deb http://archive.ubuntu.com/ubuntu noble-backports main restricted universe multiverse deb https://download.docker.com/linux/ubuntu noble stable deb https://nvidia.github.io/libnvidia-container/stable/deb/amd64 / deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64 / ``` Add DGX OS or MAAS repositories only when the selected profile uses them. Use the matching suite and architecture instead of `noble`/`amd64` for another target. Download exactly the configured set: ```bash sudo mkdir -p /var/repos sudo apt-mirror ``` Download the signing material that DeepOps fetches directly, plus the RPM bootstrap files when building an Enterprise Linux mirror. Keep these outside `mirror/` so their archive paths are deterministic: ```bash sudo mkdir -p /var/repos/keys /var/repos/bootstrap sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \ -o /var/repos/keys/docker-ubuntu.asc sudo curl -fsSL https://download.docker.com/linux/centos/gpg \ -o /var/repos/keys/docker-rpm.asc sudo curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \ -o /var/repos/keys/libnvidia-container.asc sudo curl -fsSL \ https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/3bf863cc.pub \ -o /var/repos/keys/cuda-ubuntu2404-3bf863cc.pub sudo curl -fsSL \ https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/D42D0685.pub \ -o /var/repos/keys/cuda-rhel9-D42D0685.pub sudo curl -fsSL \ https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-9 \ -o /var/repos/keys/RPM-GPG-KEY-EPEL-9 sudo curl -fsSL \ https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm \ -o /var/repos/bootstrap/epel-release-latest-9.noarch.rpm gpg --show-keys --with-fingerprint /var/repos/keys/* ``` Replace `ubuntu2404`, `rhel9`, and `9` together for another supported target. Verify the downloaded fingerprints against the vendor's published values under the site's package-trust process before transfer; do not turn off GPG checking. ### Enterprise Linux/RPM Configure the target distribution repositories plus EPEL, Docker CE, CUDA, `libnvidia-container`, and `nvidia-container-toolkit`. For EL9, download the checked-in documentation's complete additional-repository set into the top-level repo-ID directories created by `reposync`: ```bash for repo_id in epel docker-ce-stable cuda-rhel9-x86_64 \ libnvidia-container nvidia-container-toolkit; do sudo reposync -l --repoid="${repo_id}" --downloadcomps \ --download-metadata --download_path=/var/repos sudo createrepo "/var/repos/${repo_id}" done ``` Use `cuda-rhel8-x86_64` instead on EL8. Compare the final IDs and package contents with `/etc/yum.repos.d/` and the enabled workload on a connected reference host; distribution repositories remain a separate prerequisite. ## 3. Mirror direct files, charts, and images 1. Download the direct files selected by the current default Slurm profile. These names and versions come from the checked-in role defaults: ```bash sudo install -d -m 0755 /var/repos/downloads while read -r url name; do sudo curl -fL "${url}" -o "/var/repos/downloads/${name}" done <<'EOF' https://download.schedmd.com/slurm/slurm-26.05.1.tar.bz2 slurm-26.05.1.tar.bz2 https://download.open-mpi.org/release/hwloc/v2.5/hwloc-2.5.0.tar.gz hwloc-2.5.0.tar.gz https://github.com/openpmix/openpmix/releases/download/v3.2.3/pmix-3.2.3.tar.bz2 pmix-3.2.3.tar.bz2 https://github.com/mej/nhc/releases/download/1.4.3/lbnl-nhc-1.4.3.tar.xz lbnl-nhc-1.4.3.tar.xz https://github.com/NVIDIA/pyxis/archive/v0.11.1.tar.gz pyxis-0.11.1.tar.gz https://developer.download.nvidia.com/hpc-sdk/23.7/nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz https://get.helm.sh/helm-v3.17.1-linux-amd64.tar.gz helm-v3.17.1-linux-amd64.tar.gz EOF for name in slurm-26.05.1.tar.bz2 hwloc-2.5.0.tar.gz \ pmix-3.2.3.tar.bz2 lbnl-nhc-1.4.3.tar.xz pyxis-0.11.1.tar.gz \ nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz \ helm-v3.17.1-linux-amd64.tar.gz; do test -s "/var/repos/downloads/${name}" done ``` Enroot is also enabled by default, but its package filenames are owned by the pinned `nvidia.enroot` Galaxy role rather than this repository. After `./scripts/setup.sh`, use that installed role's real package list to download the target-OS packages and generate the matching Ansible override. Choose `enroot_deb_packages` for Ubuntu or `enroot_rpm_packages` for Enterprise Linux; repeat for both when the inventory mixes OS families: ```bash test -s roles/galaxy/nvidia.enroot/defaults/main.yml cat >/tmp/deepops-mirror-enroot.yml <<'EOF' --- - hosts: localhost connection: local gather_facts: true become: true vars_files: - "{{ deepops_root }}/roles/galaxy/nvidia.enroot/defaults/main.yml" tasks: - name: Download the pinned Enroot packages get_url: url: "{{ item }}" dest: "/var/repos/downloads/{{ item | basename }}" mode: "0644" loop: "{{ lookup('vars', enroot_package_list_var) }}" - name: Write the matching offline override copy: dest: "/var/repos/downloads/{{ enroot_package_list_var }}-offline.yml" mode: "0644" content: | {{ enroot_package_list_var }}: {% for package_url in lookup('vars', enroot_package_list_var) %} - "{{ offline_http_base }}/{{ package_url | basename }}" {% endfor %} EOF ansible-playbook -i localhost, /tmp/deepops-mirror-enroot.yml \ -e "deepops_root=$(pwd)" \ -e enroot_package_list_var=enroot_deb_packages \ -e offline_http_base=http://package-server/downloads ``` Replace `package-server` before running the generator. Verify every file is nonempty. Use these exact overrides with the published paths: ```yaml slurm_src_url: "http://package-server/downloads/slurm-26.05.1.tar.bz2" hwloc_src_url: "http://package-server/downloads/hwloc-2.5.0.tar.gz" pmix_src_url: "http://package-server/downloads/pmix-3.2.3.tar.bz2" nhc_src_url: "http://package-server/downloads/lbnl-nhc-1.4.3.tar.xz" slurm_pyxis_tarball_url: "http://package-server/downloads/pyxis-0.11.1.tar.gz" hpcsdk_download_url: "http://package-server/downloads/nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz" ``` Add the generated Enroot YAML list to `config/group_vars/all.yml`. On Enterprise Linux, also override `epel_package` and `epel_key_url`; several default roles import the EPEL key directly, so mirroring packages alone is insufficient. DCGM needs no separate file mirror: the `nvidia_dcgm` role installs `dcgm_pkg_name` (`datacenter-gpu-manager`) from the CUDA package repository, so it is covered by the CUDA repository mirror below — verify the DCGM package is present in the mirrored CUDA repo content. The default Slurm configuration also enables CUDA installation, whose Ubuntu and Enterprise Linux tasks fetch the CUDA repository and its signing material from `developer.download.nvidia.com`. Mirroring the CUDA packages alone does not redirect these tasks. On Ubuntu, the role uses `nvidia_driver_ubuntu_cuda_keyring_url` only to install the keyring package; it does **not** use `nvidia_driver_ubuntu_cuda_repo_baseurl` to write an APT source. The keyring package writes an upstream source that must be replaced on every target as shown in step 5. On Enterprise Linux, override both repository variables after mirroring the metadata and GPG key: - `nvidia_driver_ubuntu_cuda_keyring_url` (Ubuntu, plus the target-side APT source replacement in step 5) - `nvidia_driver_rhel_cuda_repo_baseurl` and `nvidia_driver_rhel_cuda_repo_gpgkey` (Enterprise Linux) Either mirror and override every enabled dependency or explicitly disable the optional component. Do not silently fall back to its public URL. 2. Mirror the Helm repositories/charts required by the selected Kubernetes profile. The default tree references the stable Helm repository, GPU Operator `v26.3.3`, and, when enabled, NFS subdir external provisioner `4.0.18`. Set `gpu_operator_helm_repo` and `k8s_nfs_client_helm_repo` to internal chart repositories. Archive the chart versions selected by current DeepOps defaults on the connected side: ```bash mkdir -p /tmp/charts helm repo add nvidia https://helm.ngc.nvidia.com/nvidia helm pull nvidia/gpu-operator --version v26.3.3 --destination /tmp/charts helm repo add nfs-subdir-external-provisioner \ https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner helm pull nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \ --version 4.0.18 --destination /tmp/charts helm repo index /tmp/charts ``` Omit the NFS chart only when `k8s_nfs_client_provisioner: false`. Publish `/tmp/charts` from an internal HTTP service and set the two repository variables to that URL. 3. Generate and collect the exact Kubespray file and image set from the pinned submodule. `generate_list.yml` runs only against `localhost`; passing `-i config/inventory` alone does **not** apply variables from DeepOps' `k8s_cluster` group. Export the effective variables for one representative Kubernetes host, review the selected versions, and pass that JSON as extra-vars so the localhost play uses the deployment's resolved overrides. Kubernetes download versions must be cluster-wide; stop if control-plane and worker hosts resolve different artifact-selection variables. Then render the enabled DeepOps charts with the same values used by their roles and append their images to Kubespray's list before creating the image archive: ```bash task_deepops_root=$(pwd) task_offline_dir="${task_deepops_root}/submodules/kubespray/contrib/offline" task_kube_host=kube01 # replace with one real kube_control_plane or kube_node host ansible-inventory -i config/inventory --host "${task_kube_host}" \ > /tmp/kubespray-effective-vars.json python3 - /tmp/kubespray-effective-vars.json <<'PY' import json import sys variables = json.load(open(sys.argv[1], encoding="utf-8")) for name in ("kube_version", "container_manager", "etcd_deployment_type"): print(f"{name}={variables.get(name, '<Kubespray default>')}") PY cd "${task_offline_dir}" ./generate_list.sh -i "${task_deepops_root}/config/inventory" \ -e "@/tmp/kubespray-effective-vars.json" test -s temp/files.list test -s temp/images.list NO_HTTP_SERVER=1 ./manage-offline-files.sh helm template nvidia-gpu-operator \ /tmp/charts/gpu-operator-v26.3.3.tgz \ --namespace gpu-operator \ --set driver.version=580.126.20 \ --set mig.strategy=mixed \ --set driver.enabled=false \ --set toolkit.enabled=true \ --set dcgm.enabled=false \ --set migManager.enabled=true \ > /tmp/gpu-operator-rendered.yml helm template nfs-subdir-external-provisioner \ /tmp/charts/nfs-subdir-external-provisioner-4.0.18.tgz \ > /tmp/nfs-provisioner-rendered.yml python3 - /tmp/gpu-operator-rendered.yml \ /tmp/nfs-provisioner-rendered.yml <<'PY' >> temp/images.list import sys import yaml images = set() def walk(value): if isinstance(value, dict): repository = value.get("repository") image = value.get("image")