# Deploy Airgapped

> Prepare mirrors and transfer artifacts, configure DeepOps, deploy Slurm or Kubernetes GPU clusters without Internet access, and validate them with machine-readable gates. Use for disconnected, restricted-egress, offline, or air-gapped DeepOps installations and for diagnosing missing package, file, chart, or container artifacts.

- Skill: `nvidia/deploy-airgapped` (Agent Skill)
- Install (CLI): `npx get-skill add nvidia/deploy-airgapped`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia/deploy-airgapped/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: NVIDIA (https://skillmd.com/u/nvidia)
- Updated: 2026-11-18
- Page: https://skillmd.com/skills/nvidia/deploy-airgapped

---


# Deploy DeepOps in an air-gapped environment

## Preconditions and boundaries

- Use an Internet-connected staging machine and a separate provisioning machine
  that can reach every target and every internal mirror. Do not connect an
  isolated network to the Internet for this procedure.
- Fix one DeepOps commit, target OS release, CPU architecture, cluster type,
  inventory, and enabled feature set before mirroring. Repository and image
  requirements change when any of these inputs change.
- Provide complete distribution package mirrors plus internal HTTP package/file
  service and container registry endpoints. Use trusted TLS where possible; if
  the registry is intentionally insecure, configure it explicitly.
- Bootstrap the isolated provisioning machine with the OS packages and Python
  environment required by `scripts/setup.sh` before closing the boundary, or
  provide approved internal OS and Python package indexes. Copying the source
  tree alone does not install Ansible.
- Reserve enough storage for repository metadata, packages, container images,
  charts, the initialized DeepOps checkout, and two copies during transfer.
- Confirm that every target is fully manageable and has no active users or
  workloads. Driver installation can reboot hosts.
- Run DeepOps commands from the repository root. Keep secrets out of archives
  and store site configuration in `config/`, never `config.example/`.

DeepOps does not provide supported mirror-building automation. Do **not** run
`scripts/airgap/build_offline_cache.sh`: it calls the removed
`playbooks/airgap/build-offline-cache.yml` and cannot complete in this tree.
Build and verify the mirrors explicitly as described below.

## 1. Freeze the deployment inputs

1. Record the DeepOps commit and initialize every submodule on the connected
   staging machine:

   ```bash
   git rev-parse HEAD
   git submodule update --init --recursive
   ./scripts/setup.sh
   ```

2. Copy `config.example/` to `config/`. Define the real hosts in
   `config/inventory` before collecting artifacts so the selected cluster path
   and optional components are known:

   - Slurm: controller/login hosts in `[slurm-master]`, compute hosts in
     `[slurm-node]`; a single host may be in both groups.
   - Kubernetes: control-plane hosts in `[kube_control_plane]` and `[etcd]`,
     workers in `[kube_node]`; a single host may be in all three groups.

3. Review the resolved inventory. Stop if it contains any unintended host:

   ```bash
   ansible-inventory -i config/inventory --list
   ```

4. Record the target OS release/architecture and all enabled roles. Mirror
   every dependency of that exact profile. In particular, account for:

   - distribution, Docker CE, NVIDIA CUDA/driver, NVIDIA Container Toolkit,
     and EPEL repositories where applicable;
   - direct-download archives referenced by enabled role defaults;
   - every container image and Helm chart used by the chosen Slurm or
     Kubernetes path;
   - the initialized Kubespray submodule, Galaxy roles/collections, and Python
     packages needed on the provisioning machine.

## 2. Build package mirrors on the connected side

### Ubuntu/APT

Install `apt-mirror`, set `base_path` in `/etc/apt/mirror.list` (for example,
`/var/repos`), and add the suites for the target release. For Ubuntu 24.04
(`noble`), the current DeepOps documentation uses these sources:

```text
deb http://archive.ubuntu.com/ubuntu noble main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu noble-security main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu noble-updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu noble-backports main restricted universe multiverse
deb https://download.docker.com/linux/ubuntu noble stable
deb https://nvidia.github.io/libnvidia-container/stable/deb/amd64 /
deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64 /
```

Add DGX OS or MAAS repositories only when the selected profile uses them. Use
the matching suite and architecture instead of `noble`/`amd64` for another
target. Download exactly the configured set:

```bash
sudo mkdir -p /var/repos
sudo apt-mirror
```

Download the signing material that DeepOps fetches directly, plus the RPM
bootstrap files when building an Enterprise Linux mirror. Keep these outside
`mirror/` so their archive paths are deterministic:

```bash
sudo mkdir -p /var/repos/keys /var/repos/bootstrap
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  -o /var/repos/keys/docker-ubuntu.asc
sudo curl -fsSL https://download.docker.com/linux/centos/gpg \
  -o /var/repos/keys/docker-rpm.asc
sudo curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  -o /var/repos/keys/libnvidia-container.asc
sudo curl -fsSL \
  https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/3bf863cc.pub \
  -o /var/repos/keys/cuda-ubuntu2404-3bf863cc.pub
sudo curl -fsSL \
  https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/D42D0685.pub \
  -o /var/repos/keys/cuda-rhel9-D42D0685.pub
sudo curl -fsSL \
  https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-9 \
  -o /var/repos/keys/RPM-GPG-KEY-EPEL-9
sudo curl -fsSL \
  https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm \
  -o /var/repos/bootstrap/epel-release-latest-9.noarch.rpm
gpg --show-keys --with-fingerprint /var/repos/keys/*
```

Replace `ubuntu2404`, `rhel9`, and `9` together for another supported target.
Verify the downloaded fingerprints against the vendor's published values under
the site's package-trust process before transfer; do not turn off GPG checking.

### Enterprise Linux/RPM

Configure the target distribution repositories plus EPEL, Docker CE, CUDA,
`libnvidia-container`, and `nvidia-container-toolkit`. For EL9, download the
checked-in documentation's complete additional-repository set into the
top-level repo-ID directories created by `reposync`:

```bash
for repo_id in epel docker-ce-stable cuda-rhel9-x86_64 \
  libnvidia-container nvidia-container-toolkit; do
  sudo reposync -l --repoid="${repo_id}" --downloadcomps \
    --download-metadata --download_path=/var/repos
  sudo createrepo "/var/repos/${repo_id}"
done
```

Use `cuda-rhel8-x86_64` instead on EL8. Compare the final IDs and package
contents with `/etc/yum.repos.d/` and the enabled workload on a connected
reference host; distribution repositories remain a separate prerequisite.

## 3. Mirror direct files, charts, and images

1. Download the direct files selected by the current default Slurm profile.
   These names and versions come from the checked-in role defaults:

   ```bash
   sudo install -d -m 0755 /var/repos/downloads
   while read -r url name; do
     sudo curl -fL "${url}" -o "/var/repos/downloads/${name}"
   done <<'EOF'
   https://download.schedmd.com/slurm/slurm-26.05.5.tar.bz2 slurm-26.05.5.tar.bz2
   https://download.open-mpi.org/release/hwloc/v2.5/hwloc-2.5.0.tar.gz hwloc-2.5.0.tar.gz
   https://github.com/openpmix/openpmix/releases/download/v3.2.3/pmix-3.2.3.tar.bz2 pmix-3.2.3.tar.bz2
   https://github.com/mej/nhc/releases/download/1.4.3/lbnl-nhc-1.4.3.tar.xz lbnl-nhc-1.4.3.tar.xz
   https://github.com/NVIDIA/pyxis/archive/v0.11.1.tar.gz pyxis-0.11.1.tar.gz
   https://developer.download.nvidia.com/hpc-sdk/23.7/nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz
   https://get.helm.sh/helm-v3.17.1-linux-amd64.tar.gz helm-v3.17.1-linux-amd64.tar.gz
   EOF
   for name in slurm-26.05.5.tar.bz2 hwloc-2.5.0.tar.gz \
     pmix-3.2.3.tar.bz2 lbnl-nhc-1.4.3.tar.xz pyxis-0.11.1.tar.gz \
     nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz \
     helm-v3.17.1-linux-amd64.tar.gz; do
     test -s "/var/repos/downloads/${name}"
   done
   ```

   Enroot is also enabled by default, but its package filenames are owned by
   the pinned `nvidia.enroot` Galaxy role rather than this repository. After
   `./scripts/setup.sh`, use that installed role's real package list to
   download the target-OS packages and generate the matching Ansible override.
   Choose `enroot_deb_packages` for Ubuntu or `enroot_rpm_packages` for
   Enterprise Linux; repeat for both when the inventory mixes OS families:

   ```bash
   test -s roles/galaxy/nvidia.enroot/defaults/main.yml
   cat >/tmp/deepops-mirror-enroot.yml <<'EOF'
   ---
   - hosts: localhost
     connection: local
     gather_facts: true
     become: true
     vars_files:
       - "{{ deepops_root }}/roles/galaxy/nvidia.enroot/defaults/main.yml"
     tasks:
       - name: Download the pinned Enroot packages
         get_url:
           url: "{{ item }}"
           dest: "/var/repos/downloads/{{ item | basename }}"
           mode: "0644"
         loop: "{{ lookup('vars', enroot_package_list_var) }}"
       - name: Write the matching offline override
         copy:
           dest: "/var/repos/downloads/{{ enroot_package_list_var }}-offline.yml"
           mode: "0644"
           content: |
             {{ enroot_package_list_var }}:
             {% for package_url in lookup('vars', enroot_package_list_var) %}
               - "{{ offline_http_base }}/{{ package_url | basename }}"
             {% endfor %}
   EOF
   ansible-playbook -i localhost, /tmp/deepops-mirror-enroot.yml \
     -e "deepops_root=$(pwd)" \
     -e enroot_package_list_var=enroot_deb_packages \
     -e offline_http_base=http://package-server/downloads
   ```

   Replace `package-server` before running the generator. Verify every file is
   nonempty. Use these exact overrides with the published paths:

   ```yaml
   slurm_src_url: "http://package-server/downloads/slurm-26.05.5.tar.bz2"
   hwloc_src_url: "http://package-server/downloads/hwloc-2.5.0.tar.gz"
   pmix_src_url: "http://package-server/downloads/pmix-3.2.3.tar.bz2"
   nhc_src_url: "http://package-server/downloads/lbnl-nhc-1.4.3.tar.xz"
   slurm_pyxis_tarball_url: "http://package-server/downloads/pyxis-0.11.1.tar.gz"
   hpcsdk_download_url: "http://package-server/downloads/nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz"
   ```

   Add the generated Enroot YAML list to `config/group_vars/all.yml`. On
   Enterprise Linux, also override `epel_package` and `epel_key_url`; several
   default roles import the EPEL key directly, so mirroring packages alone is
   insufficient.

   DCGM needs no separate file mirror: the `nvidia_dcgm` role installs
   `dcgm_pkg_name` (`datacenter-gpu-manager`) from the CUDA package
   repository, so it is covered by the CUDA repository mirror below — verify
   the DCGM package is present in the mirrored CUDA repo content.

   The default Slurm configuration also enables CUDA installation, whose
   Ubuntu and Enterprise Linux tasks fetch the CUDA repository and its
   signing material from `developer.download.nvidia.com`. Mirroring the CUDA
   packages alone does not redirect these tasks. On Ubuntu, the role uses
   `nvidia_driver_ubuntu_cuda_keyring_url` only to install the keyring package;
   it does **not** use `nvidia_driver_ubuntu_cuda_repo_baseurl` to write an APT
   source. The keyring package writes an upstream source that must be replaced
   on every target as shown in step 5. On Enterprise Linux, override both
   repository variables after mirroring the metadata and GPG key:

   - `nvidia_driver_ubuntu_cuda_keyring_url` (Ubuntu, plus the target-side APT
     source replacement in step 5)
   - `nvidia_driver_rhel_cuda_repo_baseurl` and
     `nvidia_driver_rhel_cuda_repo_gpgkey` (Enterprise Linux)

   Either mirror and override every enabled dependency or explicitly disable
   the optional component. Do not silently fall back to its public URL.

2. Mirror the Helm repositories/charts required by the selected Kubernetes
   profile. The default tree references the stable Helm repository, GPU
   Operator `v26.7.2`, and, when enabled, NFS subdir external provisioner
   `4.0.18`. Set `gpu_operator_helm_repo` and
   `k8s_nfs_client_helm_repo` to internal chart repositories.

   Archive the chart versions selected by current DeepOps defaults on the
   connected side:

   ```bash
   mkdir -p /tmp/charts
   helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
   helm pull nvidia/gpu-operator --version v26.7.2 --destination /tmp/charts
   helm repo add nfs-subdir-external-provisioner \
     https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner
   helm pull nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
     --version 4.0.18 --destination /tmp/charts
   helm repo index /tmp/charts
   ```

   Omit the NFS chart only when `k8s_nfs_client_provisioner: false`. Publish
   `/tmp/charts` from an internal HTTP service and set the two repository
   variables to that URL.

3. Generate and collect the exact Kubespray file and image set from the pinned
   submodule. `generate_list.yml` runs only against `localhost`; passing
   `-i config/inventory` alone does **not** apply variables from DeepOps'
   `k8s_cluster` group. Export the effective variables for one representative
   Kubernetes host, review the selected versions, and pass that JSON as
   extra-vars so the localhost play uses the deployment's resolved overrides.
   Kubernetes download versions must be cluster-wide; stop if control-plane and
   worker hosts resolve different artifact-selection variables. Then render the
   enabled DeepOps charts with the same values used by their roles and append
   their images to Kubespray's list before creating the image archive:

   ```bash
   task_deepops_root=$(pwd)
   task_offline_dir="${task_deepops_root}/submodules/kubespray/contrib/offline"
   task_kube_host=kube01 # replace with one real kube_control_plane or kube_node host
   ansible-inventory -i config/inventory --host "${task_kube_host}" \
     > /tmp/kubespray-effective-vars.json
   python3 - /tmp/kubespray-effective-vars.json <<'PY'
   import json
   import sys

   variables = json.load(open(sys.argv[1], encoding="utf-8"))
   for name in ("kube_version", "container_manager", "etcd_deployment_type"):
       print(f"{name}={variables.get(name, '<Kubespray default>')}")
   PY
   cd "${task_offline_dir}"
   ./generate_list.sh -i "${task_deepops_root}/config/inventory" \
     -e "@/tmp/kubespray-effective-vars.json"
   test -s temp/files.list
   test -s temp/images.list
   NO_HTTP_SERVER=1 ./manage-offline-files.sh

   helm template nvidia-gpu-operator \
     /tmp/charts/gpu-operator-v26.7.2.tgz \
     --namespace gpu-operator \
     --set driver.version=580.126.20 \
     --set mig.strategy=mixed \
     --set driver.enabled=false \
     --set toolkit.enabled=true \
     --set dcgm.enabled=false \
     --set migManager.enabled=true \
     > /tmp/gpu-operator-rendered.yml
   helm template nfs-subdir-external-provisioner \
     /tmp/charts/nfs-subdir-external-provisioner-4.0.18.tgz \
     > /tmp/nfs-provisioner-rendered.yml
   python3 - /tmp/gpu-operator-rendered.yml \
     /tmp/nfs-provisioner-rendered.yml <<'PY' >> temp/images.list
   import sys
   import yaml

   images = set()

   def walk(value):
       if isinstance(value, dict):
           repository = value.get("repository")
           image = value.get("image")
           version = value.get("version", value.get("tag"))
           if isinstance(repository, str) and isinstance(image, str) and version:
               reference = f"{repository.rstrip('/')}/{image.lstrip('/')}"
               if "@" not in reference and ":" not in reference.rsplit("/", 1)[-1]:
                   separator = "@" if str(version).startswith("sha256:") else ":"
                   reference = f"{reference}{separator}{version}"
               images.add(reference)
           if isinstance(image, str):
               tail = image.rsplit("/", 1)[-1]
               if "@" in image or ":" in tail:
                   images.add(image)
           for child in value.values():
               walk(child)
       elif isinstance(value, list):
           for child in value:
               walk(child)

   for manifest_path in sys.argv[1:]:
       with open(manifest_path, encoding="utf-8") as stream:
           for document in yaml.safe_load_all(stream):
               walk(document)

   if not images:
       raise SystemExit("no chart images found")
   print("\n".join(sorted(images)))
   PY
   sort -u -o temp/images.list temp/images.list
   test -z "$(grep -Ev \
     '^[^[:space:]]+(:[^/[:space:]]+|@sha256:[0-9a-f]+)$' \
     temp/images.list)"

   PRIVATE_REGISTRY=nvcr.io IMAGES_FROM_FILE="${task_offline_dir}/temp/images.list" \
     ./manage-offline-container-images.sh create
   install -d /tmp/kubespray-offline
   cp offline-files.tar.gz container-images.tar.gz temp/files.list \
     temp/images.list /tmp/kubespray-offline/
   cd "${task_deepops_root}"
   ```

   The `PRIVATE_REGISTRY=nvcr.io` setting makes the checked-in image manager
   strip `nvcr.io/` just as it strips the Kubespray source registries, so an
   internal registry mirror can serve the paths that the GPU Operator chart
   requests. If NFS is disabled, omit its `helm template` command. If site
   overrides change any GPU Operator flag, render with those values instead.
   The inventory path is still required by the generator, but the explicit
   effective-variable JSON is what makes group/host version overrides visible
   to its localhost play. Treat a missing representative host, an empty list, a
   failed pull, or an unparseable rendered image as a collection failure; do not
   continue with a partial archive.

4. Pull and archive the CUDA validator plus the images used by enabled Slurm
   monitoring/registry roles. For the current validator and registry images:

   ```bash
   mkdir -p /tmp/images
   docker pull nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
   docker save -o /tmp/images/nvidia-cuda-12.4.1-base-ubuntu22.04.tar \
     nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
   docker pull registry:3.1.2
   docker save -o /tmp/images/registry-3.1.2.tar registry:3.1.2
   ```

   When Slurm monitoring stays enabled, also mirror the images selected by
   `prometheus_container`, `grafana_container`, `alertmanager_container`,
   `node_exporter_container`, and `nvidia_dcgm_container`, then override those
   variables with their internal registry paths. The Slurm exporter is a
   special case: its default `slurm_exporter_build_image: true` builds from
   `golang:1.24` and `ubuntu:24.04`, and the runtime stage runs `apt-get`.
   Build it while connected, archive it with the other images, and use the
   prebuilt image offline:

   ```bash
   for image in prom/prometheus:v3.15.0 grafana/grafana:13.2.4 \
     prom/alertmanager:v0.34.1 quay.io/prometheus/node-exporter:v1.12.1 \
     nvcr.io/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless; do
     docker pull "${image}"
   done
   docker save -o /tmp/images/slurm-monitoring.tar \
     prom/prometheus:v3.15.0 grafana/grafana:13.2.4 \
     prom/alertmanager:v0.34.1 quay.io/prometheus/node-exporter:v1.12.1 \
     nvcr.io/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless
   docker build -t deepops/prometheus-slurm-exporter:2.0.0 \
     roles/prometheus-slurm-exporter/files/docker
   docker save -o /tmp/images/prometheus-slurm-exporter-2.0.0.tar \
     deepops/prometheus-slurm-exporter:2.0.0
   ```

   This connected-side build obtains the two base images and runtime APT
   packages. Do not leave the default local build enabled in the isolated
   environment unless an approved internal build setup supplies both base
   images and APT sources.

5. Archive the initialized checkout separately from site secrets. **Keep
   `.git` in the archive**: `playbooks/k8s-cluster.yml` unconditionally runs
   `git submodule update --init` from the repository root, so an extracted
   tree without Git metadata fails before Kubespray starts. With submodules
   already initialized at the pinned commits, that task is an offline no-op.
   Run this from the checkout's parent directory after `setup.sh` has
   populated its Galaxy dependencies and
   `git submodule update --init --recursive` has completed:

   ```bash
   git -C deepops submodule status --recursive   # every line must start with a space (initialized, pinned)
   tar --exclude='deepops/config' \
     -czf /tmp/deepops-source.tar.gz deepops
   ```

   The archive is larger with Git metadata included; that is the price of the
   Kubernetes path working. Excluding `deepops/config` still keeps site
   secrets out of the transfer artifact.

6. Package repositories and images for approved removable-media transfer and
   create checksums. Substitute a protected, user-owned staging directory for
   `/path/to/staging`; create the ISO files without elevated privileges:

   ```bash
   genisoimage -o /path/to/staging/packages.iso /var/repos
   genisoimage -o /path/to/staging/images.iso /tmp/images
   genisoimage -o /path/to/staging/charts.iso /tmp/charts
   genisoimage -o /path/to/staging/kubespray-offline.iso /tmp/kubespray-offline
   genisoimage -o /path/to/staging/keys.iso /var/repos/keys
   cp /tmp/deepops-source.tar.gz /path/to/staging/
   cd /path/to/staging
   sha256sum packages.iso images.iso charts.iso kubespray-offline.iso keys.iso \
     deepops-source.tar.gz > SHA256SUMS
   ```

Transfer the ISO files, checkout archive, and checksum file through the site's
approved boundary process. Verify the checksums again
inside the isolated network before importing anything:

```bash
cd /path/to/import
sha256sum -c SHA256SUMS
```

## 4. Import artifacts on the isolated side

1. Mount the verified transfer images and copy their contents to persistent or
   working directories:

   ```bash
   sudo mkdir -p /mnt/deepops-packages /mnt/deepops-images \
     /mnt/deepops-charts /mnt/deepops-kubespray /mnt/deepops-keys
   sudo mount -o loop /path/to/import/packages.iso /mnt/deepops-packages
   sudo mount -o loop /path/to/import/images.iso /mnt/deepops-images
   sudo mount -o loop /path/to/import/charts.iso /mnt/deepops-charts
   sudo mount -o loop /path/to/import/kubespray-offline.iso /mnt/deepops-kubespray
   sudo mount -o loop /path/to/import/keys.iso /mnt/deepops-keys
   sudo mkdir -p /var/repos /var/www/html/charts
   mkdir -p /tmp/images
   sudo cp -a /mnt/deepops-packages/. /var/repos/
   cp -a /mnt/deepops-images/. /tmp/images/
   sudo cp -a /mnt/deepops-charts/. /var/www/html/charts/
   mkdir -p /tmp/kubespray-offline
   cp -a /mnt/deepops-kubespray/. /tmp/kubespray-offline/
   ```

2. Publish signing keys first, and fail if any expected file is missing. This
   EL9/Ubuntu 24.04 example corresponds exactly to the connected-side commands:

   ```bash
   sudo install -d /var/www/html/keys
   for key in docker-ubuntu.asc docker-rpm.asc libnvidia-container.asc \
     cuda-ubuntu2404-3bf863cc.pub cuda-rhel9-D42D0685.pub \
     RPM-GPG-KEY-EPEL-9; do
     test -s "/mnt/deepops-keys/${key}"
     sudo install -m 0644 "/mnt/deepops-keys/${key}" "/var/www/html/keys/${key}"
   done
   gpg --show-keys --with-fingerprint /var/www/html/keys/*
   sha256sum /var/www/html/keys/*
   ```

   For APT, preserve `apt-mirror`'s hostname and path layout verbatim:

   ```bash
   sudo install -d /var/www/html/repos
   test -d /var/repos/mirror/archive.ubuntu.com/ubuntu/dists/noble
   test -d /var/repos/mirror/download.docker.com/linux/ubuntu/dists/noble
   test -d /var/repos/mirror/nvidia.github.io/libnvidia-container/stable/deb/amd64
   test -d /var/repos/mirror/developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64
   sudo cp -a /var/repos/mirror/. /var/www/html/repos/
   ```

   Publish the direct Slurm/Enroot files under the paths used by the generated
   overrides:

   ```bash
   test -d /var/repos/downloads
   sudo install -d /var/www/html/downloads
   sudo cp -a /var/repos/downloads/. /var/www/html/downloads/
   test -z "$(find /var/www/html/downloads -maxdepth 1 -type f -size 0 -print -quit)"
   ```

   For RPM, publish the exact top-level repo-ID directories, EPEL bootstrap
   RPM, and a rewritten Container Toolkit repo file whose base URLs are
   internal:

   ```bash
   for repo_id in epel docker-ce-stable cuda-rhel9-x86_64 \
     libnvidia-container nvidia-container-toolkit; do
     test -d "/var/repos/${repo_id}/repodata"
     sudo cp -a "/var/repos/${repo_id}" /var/www/html/repos/
   done
   test -s /var/repos/bootstrap/epel-release-latest-9.noarch.rpm
   sudo install -m 0644 /var/repos/bootstrap/epel-release-latest-9.noarch.rpm \
     /var/www/html/repos/epel-release-latest-9.noarch.rpm
   sudo tee /var/www/html/repos/nvidia-container-toolkit.repo >/dev/null <<'EOF'
   [libnvidia-container]
   name=libnvidia-container
   baseurl=http://package-server/repos/libnvidia-container
   enabled=1
   gpgcheck=1
   gpgkey=http://package-server/keys/libnvidia-container.asc

   [nvidia-container-toolkit]
   name=nvidia-container-toolkit
   baseurl=http://package-server/repos/nvidia-container-toolkit
   enabled=1
   gpgcheck=1
   gpgkey=http://package-server/keys/libnvidia-container.asc
   EOF
   ```

   Replace `package-server` with the real internal DNS name in both the repo
   file and step 5. Verify every configured file URL with `curl -fsS` and every
   APT/RPM metadata path with the target package manager before deployment.
   Preserve GPG verification.

3. Load the registry image and start the isolated registry if the site does
   not already provide one. Run this fallback on the registry host itself. It
   is plain HTTP, so the imports below push through Docker's IPv4 loopback
   exception (`127.0.0.1:5000`); cluster nodes use the reachable
   `registry-host:5000` name configured later. Do not push to
   `registry-host:5000` from the importer unless that Docker daemon has been
   separately and deliberately configured for the insecure endpoint:

   ```bash
   docker load -i /tmp/images/registry-3.1.2.tar
   docker volume create registry-images
   docker run -d -p 5000:5000 --restart=always --name registry \
     -v registry-images:/var/lib/registry registry:3.1.2
   ```

4. Import, retag, and push every archived image on that registry host. Preserve
   the complete repository path expected by the consuming chart or role. The
   hostname used for the push is not part of the stored repository path, so a
   cluster pull of `registry-host:5000/nvidia/cuda:...` resolves the image
   pushed as `127.0.0.1:5000/nvidia/cuda:...`:

   ```bash
   docker load -i /tmp/images/nvidia-cuda-12.4.1-base-ubuntu22.04.tar
   docker tag nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04 \
     127.0.0.1:5000/nvidia/cuda:12.4.1-base-ubuntu22.04
   docker push 127.0.0.1:5000/nvidia/cuda:12.4.1-base-ubuntu22.04
   docker load -i /tmp/images/prometheus-slurm-exporter-2.0.0.tar
   docker tag deepops/prometheus-slurm-exporter:2.0.0 \
     127.0.0.1:5000/deepops/prometheus-slurm-exporter:2.0.0
   docker push 127.0.0.1:5000/deepops/prometheus-slurm-exporter:2.0.0
   docker load -i /tmp/images/slurm-monitoring.tar
   while read -r source target; do
     docker tag "${source}" "${target}"
     docker push "${target}"
   done <<'EOF'
   prom/prometheus:v3.15.0 127.0.0.1:5000/prom/prometheus:v3.15.0
   grafana/grafana:13.2.4 127.0.0.1:5000/grafana/grafana:13.2.4
   prom/alertmanager:v0.34.1 127.0.0.1:5000/prom/alertmanager:v0.34.1
   quay.io/prometheus/node-exporter:v1.12.1 127.0.0.1:5000/prometheus/node-exporter:v1.12.1
   nvcr.io/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless 127.0.0.1:5000/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless
   EOF
   ```

5. Extract the initialized DeepOps checkout on the provisioning machine:

   ```bash
   mkdir -p "$HOME/deepops-src"
   tar -xzf /path/to/import/deepops-source.tar.gz -C "$HOME/deepops-src"
   cd "$HOME/deepops-src/deepops"
   ```

   Activate the provisioning environment prepared before isolation (or install
   it from approved internal indexes), then copy and edit `config.example/` if
   site configuration was not transferred separately. Install the mirrored
   Helm version expected by `scripts/k8s/install_helm.sh` on an amd64 Linux
   provisioning host so that script does not fetch its public installer:

   ```bash
   mkdir -p /tmp/deepops-helm
   tar -xzf /var/repos/downloads/helm-v3.17.1-linux-amd64.tar.gz \
     -C /tmp/deepops-helm
   sudo install -m 0755 /tmp/deepops-helm/linux-amd64/helm /usr/local/bin/helm
   helm version --short | grep '^v3.17.1'
   ```

   Mirror the matching archive for another provisioning platform; do not use
   the amd64 archive there.

6. Publish the Kubespray static-file tree, then import every Kubespray, GPU
   Operator, and NFS image collected in step 3 into the registry. Do **not** run
   the pinned manager's `register` action: before importing, it overwrites
   `/etc/docker/daemon.json` or `/etc/containers/registries.conf`, substitutes
   the importer's hostname instead of `DESTINATION_REGISTRY`, and does not back
   up, restore, or restart the runtime. Extract its archive and perform the same
   load/tag/push loop without changing host configuration:

   ```bash
   sudo install -d /var/www/html/kubespray
   sudo tar -xzf /tmp/kubespray-offline/offline-files.tar.gz \
     -C /var/www/html/kubespray
   test -d /var/www/html/kubespray/offline-files

   task_kubespray_import=$(mktemp -d)
   (
     set -euo pipefail
     trap 'rm -rf "${task_kubespray_import}"' EXIT
     tar -xzf /tmp/kubespray-offline/container-images.tar.gz \
       -C "${task_kubespray_import}"
     while read -r archive repository; do
       test -n "${archive}" && test -n "${repository}"
       load_line=$(docker load \
         -i "${task_kubespray_import}/container-images/${archive}" | sed -n '1p')
       source_ref=$(printf '%s\n' "${load_line}" | awk '{print $3}')
       if [ "${source_ref}" = ID: ]; then
         source_ref=$(printf '%s\n' "${load_line}" | awk '{print $4}')
       fi
       image_id=$(docker image inspect --format '{{.Id}}' "${source_ref}")
       test -n "${image_id}"
       docker tag "${image_id}" "127.0.0.1:5000/${repository}"
       docker push "127.0.0.1:5000/${repository}"
     done < "${task_kubespray_import}/container-images/container-images.txt"
   )
   ```

   These commands assume the fallback plain-HTTP registry is local to the
   importer. For a site registry on another host, require trusted TLS or have
   the site administrator configure and restart the importer's runtime under
   site change control before substituting that registry endpoint. Confirm the
   imported repositories through the registry's normal catalog or pull checks
   before deployment.

## 5. Configure DeepOps for internal endpoints

Set mirror overrides in `config/group_vars/all.yml` (or a narrower group file).
Use values matching the paths actually published by the site:

```yaml
docker_ubuntu_repo_base_url: "http://package-server/repos/download.docker.com/linux/ubuntu"
docker_ubuntu_repo_gpgkey: "http://package-server/keys/docker-ubuntu.asc"
nvidia_container_toolkit_repo_base_url: "http://package-server/repos/nvidia.github.io/libnvidia-container"
nvidia_container_toolkit_repo_gpg_url: "http://package-server/keys/libnvidia-container.asc"

docker_rh_repo_base_url: "http://package-server/repos/docker-ce-stable"
docker_rh_repo_gpgkey: "http://package-server/keys/docker-rpm.asc"
nvidia_container_toolkit_rpm_repo_url: "http://package-server/repos/nvidia-container-toolkit.repo"

docker_insecure_registries:
  - "registry-host:5000"

# CUDA repository mirror (required by the default Slurm path; see below)
nvidia_driver_ubuntu_cuda_keyring_url: "http://package-server/repos/developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb"
nvidia_driver_rhel_cuda_repo_baseurl: "http://package-server/repos/cuda-rhel9-x86_64/"
nvidia_driver_rhel_cuda_repo_gpgkey: "http://package-server/keys/cuda-rhel9-D42D0685.pub"

# EPEL mirror (Enterprise Linux; key import fails offline without this)
epel_package: "http://package-server/repos/epel-release-latest-9.noarch.rpm"
epel_key_url: "http://package-server/keys/RPM-GPG-KEY-EPEL-9"
```

For Ubuntu 24.04, prepare **every target** before running the Slurm playbook or
any play that includes `nvidia_cuda` or `nvidia_dcgm`. Replace
`package-server` with the real internal host first. These commands install the
mirrored keyring, remove the public source that package creates, write the
internal `cuda-compute-repo.list`, update that source in isolation, and then
prove the complete APT update resolves no public NVIDIA host before allowing a
normal update:

```bash
(
  set -euo pipefail
  task_cuda_repo_url='http://package-server/repos/developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64'
  task_cuda_keyring_url="${task_cuda_repo_url}/cuda-keyring_1.1-1_all.deb"

  case "${task_cuda_repo_url}" in
    *package-server*|https://developer.download.nvidia.com/*|http://developer.download.nvidia.com/*)
      echo 'ERROR: set task_cuda_repo_url to the internal CUDA mirror' >&2
      exit 1
      ;;
  esac

  task_cuda_keyring_deb=$(mktemp)
  task_apt_uris=$(mktemp)
  trap 'rm -f "${task_cuda_keyring_deb}" "${task_apt_uris}"' EXIT
  curl -fsSL "${task_cuda_keyring_url}" -o "${task_cuda_keyring_deb}"
  test -s "${task_cuda_keyring_deb}"
  sudo dpkg -i "${task_cuda_keyring_deb}"

  if [ -e /etc/apt/sources.list.d/cuda-ubuntu2404-x86_64.list ]; then
    sudo mv --backup=numbered -- \
      /etc/apt/sources.list.d/cuda-ubuntu2404-x86_64.list \
      /etc/apt/sources.list.d/cuda-ubuntu2404-x86_64.list.disabled
  fi
  printf 'deb [signed-by=/usr/share/keyrings/cuda-archive-keyring.gpg] %s /\n' \
    "${task_cuda_repo_url%/}" |
    sudo tee /etc/apt/sources.list.d/cuda-compute-repo.list >/dev/null

  sudo apt-get \
    -o Dir::Etc::sourcelist='sources.list.d/cuda-compute-repo.list' \
    -o Dir::Etc::sourceparts='-' update

  apt-get --print-uris update | tee "${task_apt_uris}"
  task_public_nvidia_urls=$(
    sed -n "s/^'\(https\?:\/\/[^']*\)'.*/\1/p" "${task_apt_uris}" |
      awk -F/ '{ host = tolower($3); sub(/:.*/, "", host) }
                host == "developer.download.nvidia.com" ||
                host == "repo.download.nvidia.com" ||
                host == "nvidia.github.io" { print }'
  )
  if [ -n "${task_public_nvidia_urls}" ]; then
    printf 'ERROR: public NVIDIA APT URL still resolves:\n%s\n' \
      "${task_public_nvidia_urls}" >&2
    exit 1
  fi
  sudo apt-get update
)
```

Use the matching repository directory and keyring-generated source filename on
another Ubuntu release. Keep all other APT sources internal too: the final
`apt-get update` intentionally checks the complete target configuration, which
is what the role will update. Retain the
`nvidia_driver_ubuntu_cuda_keyring_url` override so the role finds the mirrored
package; do not set `nvidia_driver_ubuntu_cuda_repo_baseurl` expecting it to
redirect APT.

For Slurm, disable the default pull-through registry cache when upstream
Docker Hub is unreachable. If using the site's registry, the smallest honest
profile is:

```yaml
slurm_enable_container_registry: false
standalone_container_registry_cache_enable: false
slurm_exporter_build_image: false
slurm_exporter_container: "registry-host:5000/deepops/prometheus-slurm-exporter:2.0.0"
prometheus_container: "registry-host:5000/prom/prometheus:v3.15.0"
grafana_container: "registry-host:5000/grafana/grafana:13.2.4"
alertmanager_container: "registry-host:5000/prom/alertmanager:v0.34.1"
node_exporter_container: "registry-host:5000/prometheus/node-exporter:v1.12.1"
nvidia_dcgm_container: "registry-host:5000/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless"
```

Mirror and override the Slurm source/archive variables listed above. If their
artifacts were deliberately excluded, disable the corresponding optional
features, for example `slurm_enable_monitoring`, `slurm_install_hpcsdk`,
`slurm_install_nhc`, `slurm_install_enroot`, or `slurm_install_pyxis`. Do not
disable a required feature merely to make a playbook pass.

For Kubernetes, add the pinned Kubespray offline rewrites and the DeepOps chart
repositories. This layout matches the static-file extraction and image-manager
registration commands above:

```yaml
registry_host: "registry-host:5000"
registry_addr: "registry-host:5000"
files_repo: "http://package-server/kubespray/offline-files"

kube_image_repo: "{{ registry_host }}"
gcr_image_repo: "{{ registry_host }}"
docker_image_repo: "{{ registry_host }}"
quay_image_repo: "{{ registry_host }}"
github_image_repo: "{{ registry_host }}"
github_url: "{{ files_repo }}/github.com"
dl_k8s_io_url: "{{ files_repo }}/dl.k8s.io"
storage_googleapis_url: "{{ files_repo }}/storage.googleapis.com"
get_helm_url: "{{ files_repo }}/get.helm.sh"
local_path_provisioner_helper_image_repo: "{{ registry_host }}/busybox"

gpu_operator_helm_repo: "http://package-server/charts"
k8s_nfs_client_helm_repo: "http://package-server/charts"

containerd_registries_mirrors:
  - prefix: "{{ registry_addr }}"
    mirrors:
      - host: "http://{{ registry_addr }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true
  - prefix: "nvcr.io"
    mirrors:
      - host: "http://{{ registry_addr }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true
  - prefix: "registry.k8s.io"
    mirrors:
      - host: "http://{{ registry_addr }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true
  - prefix: "docker.io"
    mirrors:
      - host: "http://{{ registry_addr }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true
  - prefix: "quay.io"
    mirrors:
      - host: "http://{{ registry_addr }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true
  - prefix: "gcr.io"
    mirrors:
      - host: "http://{{ registry_addr }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true
  - prefix: "ghcr.io"
    mirrors:
      - host: "http://{{ registry_addr }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true
```

Use trusted TLS and `skip_verify: false` when the internal registry provides
it. Set `k8s_nfs_client_provisioner: false` only if the cluster uses a
site-owned storage path or intentionally has no dynamic NFS provisioner.

The current top-level Kubernetes playbook invokes a Helm installer URL and
adds `https://charts.helm.sh/stable`; when the Ansible host platform differs
from the cluster nodes (for example a macOS or arm64 control machine), its
artifact step also downloads `kubectl` and its checksum directly from
`https://dl.k8s.io`. Run the deployment from a control host matching the
cluster platform so the playbook fetches `kubectl` from a cluster node
instead, or pre-approve an internal mirror for that URL. Its containerd
local-registry settings
also carry an explicit TODO in `config.example/group_vars/k8s_cluster.yml`.
Therefore, before declaring a fully disconnected Kubernetes run ready, prove
that these references are satisfied by approved internal endpoints or obtain a
DeepOps code change. Do not claim that the current playbook is turnkey offline.

## 6. Preflight and deploy

Run the machine-readable doctor from the isolated provisioning machine:

```bash
python3 scripts/validation/deepops_doctor.py --remote --json
```

Require exit code `0` and top-level `"ok": true`. Review the reported host
count/groups against the approved inventory. The GPU PCI count is informational;
all other failed checks must be understood and resolved before deployment.

### Slurm path

```bash
ansible-playbook -l slurm-cluster playbooks/slurm-cluster.yml
```

Rerun the same command after a transient mirror/package error. Validate on a
Slurm controller, login, or compute node:

```bash
python3 scripts/validation/validate_slurm.py --json
```

Require exit code `0`, `"ok": true`, `"nodes_unavailable": 0`,
`"gpus_configured"` greater than zero, and `"gpu_job_ok": true`.

### Kubernetes path

Run only after all Kubernetes package, file, chart, and image references have
been resolved to the isolated environment:

```bash
ansible-playbook -l k8s_cluster playbooks/k8s-cluster.yml
```

Validate from a mac

…(truncated)
