deploy-mori

# Deploy MORI You are helping the user deploy MORI inside a Docker container. **Locate `MORI_REPO_DIR`**: default to the current working directory if it contains `pyproject.toml`; ask the user otherwise. **Detect whether `docker` needs `sudo`** — every `docker` command below is written with a `sudo` prefix, but on hosts where the calling user is already in the `docker` group, `sudo docker ...` fails non-interactively with `sudo: a password is required` (no TTY to prompt for a password). Check first: ```bash docker ps &>/dev/null && echo "docker works without sudo" || echo "docker needs sudo" ``` If it works without `sudo`, drop the `sudo` prefix from **every** `docker` command in this skill (run/exec/inspect/start/cp/ps, …) for the rest of the session. **Detect NIC type** — determines whether Step 3 is needed: ```bash lspci | grep -iE "pensando|ionic|dsc|pollara" && echo "→ ainic" lspci | grep -iE "broadcom.*thor|bnxt" && echo "→ thor2" lspci | grep -iE "mellanox|connectx|nvidia.*bluefield" && echo "→ mlx5" ``` A host can have more than one type present — that's fine, `mori check`/`mori setup` only act on the vendor with the most matching RDMA devices. For mlx5, also check RoCE vs native InfiniBand (changes what Step 3c installs): ```bash for d in /sys/class/infiniband/mlx5_*; do echo "$(basename "$d"): $(cat "$d/ports/1/link_layer" 2>/dev/null)" done ``` --- ## Step 1: Start the Docker container Use the container name from the user's args if provided; ask if not. Default image: `rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.12.0` (use unless the user specifies another) — same base the CI workflows build on. ```bash if sudo docker inspect $CONTAINER_NAME &>/dev/null; then sudo docker start $CONTAINER_NAME else sudo docker run <flags> --name $CONTAINER_NAME $IMAGE_NAME sleep infinity fi ``` Probe optional mount paths on the host, only include ones that exist: ```bash for p in /shared /apps /dev/infiniband /sys/kernel/config /sys/kernel/debug; do [ -e "$p" ] && echo "EXISTS: $p" || echo "MISSING: $p" done ``` Full `docker run` flags (omit missing paths): ```bash sudo docker run \ --group-add video \ --network=host \ --ulimit nproc=100000:100000 \ --ulimit memlock=-1:-1 \ --pids-limit=-1 \ --device=/dev/kfd \ --device=/dev/dri \ --device=/dev/infiniband \ # only if exists --ipc=host \ --privileged -d \ -v /home/:/home/ \ -v /root:/root \ -v /mnt:/mnt \ -v /shared:/shared \ # only if exists -v /apps:/apps \ # only if exists -v /lib/modules:/lib/modules \ -v /sys/kernel/config:/sys/kernel/config \ # REQUIRED for bnxt DCQCN (configfs) -v /sys/kernel/debug:/sys/kernel/debug \ # REQUIRED for bnxt DCQCN (debugfs) --rm \ --name $CONTAINER_NAME \ $IMAGE_NAME \ sleep infinity ``` Notes: - `--network=host` + `--device=/dev/infiniband` required for RDMA visibility. - `--ulimit memlock=-1:-1` — RDMA pins memory on QP creation; needed for the parallel `mori check` bandwidth mesh. - configfs/debugfs mounts are **required for bnxt** DCQCN (`mori check`/`mori setup` read/write congestion control through them). `--privileged` alone doesn't propagate them — if missing on the host: `sudo mount -t configfs none /sys/kernel/config; sudo mount -t debugfs none /sys/kernel/debug`. - `--rm` — data outside mounted volumes is lost on stop. All subsequent steps run **inside** `$CONTAINER_NAME` via `docker exec`. --- ## Step 2: Install base system packages ```bash sudo docker exec $CONTAINER_NAME bash -c "apt-get update && apt-get install -y --no-install-recommends \ git libpci-dev pciutils sudo libdw1 libibverbs-dev ibverbs-utils rdma-core \ locales iputils-ping iproute2 ethtool jq perftest \ wget unzip ca-certificates curl \ libgrpc++-dev protobuf-compiler-grpc libprotobuf-dev protobuf-compiler \ libopenmpi-dev openmpi-bin" ``` > **If `apt-get update` can't reach `archive.ubuntu.com`/`security.ubuntu.com`** > (`Network is unreachable` / `connection timed out`, often on cloud hosts > whose network only routes to a provider-internal mirror — e.g. seen on an > Oracle Cloud instance routed through `iad-ad-1.clouds.archive.ubuntu.com`): > the container's default `/etc/apt/sources.list` doesn't know about that > mirror even though the **host** does. Fix by copying the host's working > sources into the container before retrying: > ```bash > sudo docker cp /etc/apt/sources.list $CONTAINER_NAME:/etc/apt/sources.list > ``` Non-obvious package roles: - `rdma-core` — version info `mori check` Step 1 reads. Without it, Step 1 still passes but logs a cosmetic `[WARN] rdma-core package not found`. - `pciutils` (`lspci`) — `nicctl` shells out to it; missing it breaks the ionic firmware/QoS/DCQCN checks with `Invalid card handle`. - `sudo` — ionic/bnxt paths of `mori check`/`mori setup` invoke `nicctl`, `dcb`, `ethtool`, sysfs writes via `sudo`. - `perftest` — `ib_write_bw`/`ib_write_lat` for the bandwidth/latency checks (steps 4/5/6). A perftest without ROCm support covers the host-memory mesh, but the GPU-memory pass in step 5 needs a build with ROCm support (`--use_rocm`). `mori check` never builds it — if a ROCm-capable perftest is missing it warns and prints the build command, then skips the GPU-memory pass. It is needed on the **peer** as well for steps 5/6. - `iproute2` — provides `dcb`, needed by `mori setup` on bnxt. - `libgrpc++-dev` + protobuf packages — build defaults to `BUILD_UMBP=ON`, whose CMake step needs gRPC headers. (`cmake`/`ninja`/`pybind11` come from `pyproject.toml` build isolation automatically.) - `libopenmpi-dev openmpi-bin` — needed for `MORI_WITH_MPI=ON` / `BUILD_BENCHMARK=ON` (benchmarks are gated behind `WITH_MPI`). --- ## Step 3: Install NIC userspace libraries Run the subsection matching the NIC type detected above: - `ainic` → **Step 3a**, `thor2`/bnxt → **Step 3b**, `mlx5` → **Step 3c** --- ## Step 3a: Install AINIC userspace libraries (AINIC only) **Skip if NIC type is not `ainic`.** > **Recommended version**: for cross-node MORI (EP over RDMA / IBGDA), AINIC firmware > `>= 1.117.5-a-45` is solid. The `1.117.1` major does **not** support IBGDA — if the > host is on that branch, flag it to the user and recommend upgrading before proceeding. > The userspace library (`libionic`) must match the kernel driver version. ### Detect host AINIC version and check against public repo Run on the **host**: ```bash IB_DEV=$(ls /sys/class/infiniband/ 2>/dev/null | head -1) if [ -z "$IB_DEV" ]; then echo "ERROR: no InfiniBand device found under /sys/class/infiniband/" echo "Make sure the ionic kernel module is loaded on the host." exit 1 fi HOST_AINIC_VER=$(cat /sys/class/infiniband/${IB_DEV}/fw_ver 2>/dev/null) if [ -z "$HOST_AINIC_VER" ]; then echo "ERROR: cannot read fw_ver from /sys/class/infiniband/${IB_DEV}/fw_ver" exit 1 fi echo "Host AINIC firmware version: $HOST_AINIC_VER (detected from $IB_DEV)" AVAILABLE=$(curl -fsSL https://repo.radeon.com/amdainic/pensando/ubuntu/ \ | grep -oP '(?<=href=")[^"]+(?=/)' \ | grep -v '^\.\.' | grep -v '^https' | sort) echo "Available AINIC versions in public repo:" echo "$AVAILABLE" if ! echo "$AVAILABLE" | grep -qx "$HOST_AINIC_VER"; then echo "" echo "ERROR: host AINIC version '$HOST_AINIC_VER' is not available in the public repo." echo "Available versions: $(echo $AVAILABLE | tr '\n' ' ')" echo "Please contact your AINIC vendor for a matching software bundle." exit 1 fi echo "Found matching version '$HOST_AINIC_VER' in public repo — proceeding." ``` ### Install inside container ```bash sudo docker exec $CONTAINER_NAME bash -c " set -e AINIC_VERSION=$HOST_AINIC_VER UBUNTU_CODENAME=\$(. /etc/os-release && echo \"\$VERSION_CODENAME\") mkdir -p /etc/apt/keyrings curl -fsSL https://repo.radeon.com/rocm/rocm.gpg.key \ | gpg --dearmor > /etc/apt/keyrings/amdainic.gpg echo \"deb [arch=amd64 signed-by=/etc/apt/keyrings/amdainic.gpg] \ https://repo.radeon.com/amdainic/pensando/ubuntu/\${AINIC_VERSION} \ \${UBUNTU_CODENAME} main\" > /etc/apt/sources.list.d/amdainic.list apt-get update apt-get install -y nicctl libionic-dev ionic-common ldconfig ldconfig -p | grep libionic nicctl --version " ``` --- ## Step 3b: Install Broadcom (bnxt / thor2) userspace libraries + tools **Skip if NIC type is not `thor2`/bnxt.** > **Recommended version**: for cross-node MORI (EP over RDMA / IBGDA), Broadcom firmware > is solid on `237.1.137.x` (official Broadcom release) and `235.2.86.x` (customer-specific > build). Known bad: `231.x` and `232.x` — if the host is on either branch, flag it to the > user and recommend upgrading. Any other branch is unverified and treated the same way. > The userspace library (`libbnxt_re`, 3b.2) must match the kernel > driver version detected in 3b.1. The bnxt path of `mori check` / `mori setup` needs two NIC-specific pieces installed here: (1) the **RoCE userspace lib** (`libbnxt_re`) and (2) a **recent `niccli`**. (`dcb`, the third dependency, comes with `iproute2` from Step 2 — just verified in 3b.4.) ### 3b.1 — Detect host bnxt version (match the userspace lib to it) Run on the **host**: ```bash modinfo -F version bnxt_re 2>/dev/null || cat /sys/module/bnxt_re/version # e.g. 235.2.88.0 → install the closest bnxt-rocelib (e.g. 235.2.86.0) ``` ### 3b.2 — Install RoCE userspace lib via the Broadcom apt repo ```bash sudo docker exec $CONTAINER_NAME bash -c ' set -e install -m 0755 -d /etc/apt/keyrings curl -fsSL https://packages.broadcom.com/artifactory/api/security/keypair/PackagesKey/public \ -o /etc/apt/keyrings/broadcom-nic.asc chmod a+r /etc/apt/keyrings/broadcom-nic.asc echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/broadcom-nic.asc] \ https://packages.broadcom.com/artifactory/ethernet-nic-debian-public jammy main" \ > /etc/apt/sources.list.d/broadcom-nic.list apt-get update # pin to match the host bnxt_re version from 3b.1; list options: apt-cache madison bnxt-rocelib apt-get install -y ibverbs-utils bnxt-rocelib=235.2.86.0 # Make the out-of-tree provider visible on a standard path. `mori check` also # accepts the rdma-core in-tree provider (libbnxt_re-rdmav<abi>.so under a # libibverbs dir) and honours $LD_LIBRARY_PATH, so this copy is only needed to # have the *Broadcom* build take precedence and get its version reported. cp /usr/local/lib/x86_64-linux-gnu/libbnxt_re* /usr/local/lib/. ldconfig ' ``` > Replace `235.2.86.0` with the version matching the host (3b.1). If gone from > the repo, pick the nearest from `apt-cache madison bnxt-rocelib`. ### 3b.3 — Install a recent `niccli` (must support the `qos` subcommand) `mori check` Step 2 uses `niccli ... qos`, so install one tracking the firmware (236.x/237.x for BCM57608): ```bash sudo docker exec $CONTAINER_NAME bash -c ' set -e NICCLI_VER=237.1.145.0 # deb version NICCLI_PKG=237.1.148.0 # BRCM_<this> path segment in the URL URL="https://docs.broadcom.com/docs-and-downloads/ethernet-network-adapters/NXE/BRCM_${NICCLI_PKG}/niccli/Linux/niccli-${NICCLI_VER}_linux.zip" cd /tmp && wget -q -O niccli.zip "$URL" && unzip -o -q niccli.zip -d niccli dpkg -i "$(find ./niccli -name "niccli_*_x86_64.deb" | head -1)" niccli -l | head # must list all NICs niccli -i 1 qos --ingress --cosq --show | head # should print the CoSQ table (TC/State/Mode) ' ``` > Alternative (no download): host usually has a working niccli at > `/opt/niccli`. `sudo docker cp /opt/niccli $CONTAINER_NAME:/opt/` then > `docker exec $CONTAINER_NAME ln -sf /opt/niccli/niccli /usr/bin/niccli`. ### 3b.4 — `dcb` (iproute2), needed by `mori setup` ```bash sudo docker exec $CONTAINER_NAME bash -c "command -v dcb" ``` > `mori setup` configures PFC/ETS via `dcb` and DCQCN via configfs, both at > host level. With `--network=host` the container shares the host netns, so > running it in-container is equivalent — just needs `dcb` + Step 1's mounts. --- ## Step 3c: Install Mellanox/NVIDIA (mlx5) userspace libraries + tools **Skip if NIC type is not `mlx5`.**