deploy-mori

name	deploy-mori
description	Deploy and set up the MORI environment in a fresh Docker container or bare host: start the container, install ROCm dependencies, NIC userspace libraries (AINIC/Broadcom/Mellanox-NVIDIA), RDMA-core, and MORI itself. Use when the user asks to deploy MORI, install MORI in a container, set up a fresh dev environment for MORI, or prepare an AINIC / Thor2 / mlx5 box for MORI.

Deploy MORI

You are helping the user deploy MORI inside a Docker container.

Locate MORI_REPO_DIR: default to the current working directory if it contains pyproject.toml; ask the user otherwise.

Detect whether docker needs sudo — every docker command below is written with a sudo prefix, but on hosts where the calling user is already in the docker group, sudo docker ... fails non-interactively with sudo: a password is required (no TTY to prompt for a password). Check first:

docker ps &>/dev/null && echo "docker works without sudo" || echo "docker needs sudo"

If it works without sudo, drop the sudo prefix from every docker command in this skill (run/exec/inspect/start/cp/ps, …) for the rest of the session.

Detect NIC type — determines whether Step 3 is needed:

lspci | grep -iE "pensando|ionic|dsc|pollara"      && echo "→ ainic"
lspci | grep -iE "broadcom.*thor|bnxt"             && echo "→ thor2"
lspci | grep -iE "mellanox|connectx|nvidia.*bluefield" && echo "→ mlx5"

A host can have more than one type present — that's fine, mori check/mori setup only act on the vendor with the most matching RDMA devices.

For mlx5, also check RoCE vs native InfiniBand (changes what Step 3c installs):

for d in /sys/class/infiniband/mlx5_*; do
  echo "$(basename "$d"): $(cat "$d/ports/1/link_layer" 2>/dev/null)"
done

---

Step 1: Start the Docker container

Use the container name from the user's args if provided; ask if not. Default image: rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.12.0 (use unless the user specifies another) — same base the CI workflows build on.

if sudo docker inspect $CONTAINER_NAME &>/dev/null; then
  sudo docker start $CONTAINER_NAME
else
  sudo docker run <flags> --name $CONTAINER_NAME $IMAGE_NAME sleep infinity
fi

Probe optional mount paths on the host, only include ones that exist:

for p in /shared /apps /dev/infiniband /sys/kernel/config /sys/kernel/debug; do
  [ -e "$p" ] && echo "EXISTS: $p" || echo "MISSING: $p"
done

Full docker run flags (omit missing paths):

sudo docker run \
    --group-add video \
    --network=host \
    --ulimit nproc=100000:100000 \
    --ulimit memlock=-1:-1 \
    --pids-limit=-1 \
    --device=/dev/kfd \
    --device=/dev/dri \
    --device=/dev/infiniband \   # only if exists
    --ipc=host \
    --privileged -d \
    -v /home/:/home/ \
    -v /root:/root \
    -v /mnt:/mnt \
    -v /shared:/shared \         # only if exists
    -v /apps:/apps \             # only if exists
    -v /lib/modules:/lib/modules \
    -v /sys/kernel/config:/sys/kernel/config \   # REQUIRED for bnxt DCQCN (configfs)
    -v /sys/kernel/debug:/sys/kernel/debug \     # REQUIRED for bnxt DCQCN (debugfs)
    --rm \
    --name $CONTAINER_NAME \
    $IMAGE_NAME \
    sleep infinity

Notes:

• --network=host + --device=/dev/infiniband required for RDMA visibility.
• --ulimit memlock=-1:-1 — RDMA pins memory on QP creation; needed for the parallel mori check bandwidth mesh.
• configfs/debugfs mounts are required for bnxt DCQCN (mori check/mori setup read/write congestion control through them). --privileged alone doesn't propagate them — if missing on the host: sudo mount -t configfs none /sys/kernel/config; sudo mount -t debugfs none /sys/kernel/debug.
• --rm — data outside mounted volumes is lost on stop.

All subsequent steps run inside $CONTAINER_NAME via docker exec.

---

Step 2: Install base system packages

sudo docker exec $CONTAINER_NAME bash -c "apt-get update && apt-get install -y --no-install-recommends \
    git libpci-dev pciutils sudo libdw1 libibverbs-dev ibverbs-utils rdma-core \
    locales iputils-ping iproute2 ethtool jq perftest \
    wget unzip ca-certificates curl \
    libgrpc++-dev protobuf-compiler-grpc libprotobuf-dev protobuf-compiler \
    libopenmpi-dev openmpi-bin"

If apt-get update can't reach archive.ubuntu.com/security.ubuntu.com (Network is unreachable / connection timed out, often on cloud hosts whose network only routes to a provider-internal mirror — e.g. seen on an Oracle Cloud instance routed through iad-ad-1.clouds.archive.ubuntu.com): the container's default /etc/apt/sources.list doesn't know about that mirror even though the host does. Fix by copying the host's working sources into the container before retrying:

sudo docker cp /etc/apt/sources.list $CONTAINER_NAME:/etc/apt/sources.list

Non-obvious package roles:

• rdma-core — version info mori check Step 1 reads. Without it, Step 1 still passes but logs a cosmetic [WARN] rdma-core package not found.
• pciutils (lspci) — nicctl shells out to it; missing it breaks the ionic firmware/QoS/DCQCN checks with Invalid card handle.
• sudo — ionic/bnxt paths of mori check/mori setup invoke nicctl, dcb, ethtool, sysfs writes via sudo.
• perftest — ib_write_bw/ib_write_lat for the bandwidth/latency checks (steps 4/5/6). A perftest without ROCm support covers the host-memory mesh, but the GPU-memory pass in step 5 needs a build with ROCm support (--use_rocm). mori check never builds it — if a ROCm-capable perftest is missing it warns and prints the build command, then skips the GPU-memory pass. It is needed on the peer as well for steps 5/6.
• iproute2 — provides dcb, needed by mori setup on bnxt.
• libgrpc++-dev + protobuf packages — build defaults to BUILD_UMBP=ON, whose CMake step needs gRPC headers. (cmake/ninja/pybind11 come from pyproject.toml build isolation automatically.)
• libopenmpi-dev openmpi-bin — needed for MORI_WITH_MPI=ON / BUILD_BENCHMARK=ON (benchmarks are gated behind WITH_MPI).

---

Step 3: Install NIC userspace libraries

Run the subsection matching the NIC type detected above:

• ainic → Step 3a, thor2/bnxt → Step 3b, mlx5 → Step 3c

---

Step 3a: Install AINIC userspace libraries (AINIC only)

Skip if NIC type is not ainic.

Recommended version: for cross-node MORI (EP over RDMA / IBGDA), AINIC firmware >= 1.117.5-a-45 is solid. The 1.117.1 major does not support IBGDA — if the host is on that branch, flag it to the user and recommend upgrading before proceeding. The userspace library (libionic) must match the kernel driver version.

Detect host AINIC version and check against public repo

Run on the host:

IB_DEV=$(ls /sys/class/infiniband/ 2>/dev/null | head -1)
if [ -z "$IB_DEV" ]; then
  echo "ERROR: no InfiniBand device found under /sys/class/infiniband/"
  echo "Make sure the ionic kernel module is loaded on the host."
  exit 1
fi
HOST_AINIC_VER=$(cat /sys/class/infiniband/${IB_DEV}/fw_ver 2>/dev/null)
if [ -z "$HOST_AINIC_VER" ]; then
  echo "ERROR: cannot read fw_ver from /sys/class/infiniband/${IB_DEV}/fw_ver"
  exit 1
fi
echo "Host AINIC firmware version: $HOST_AINIC_VER (detected from $IB_DEV)"

AVAILABLE=$(curl -fsSL https://repo.radeon.com/amdainic/pensando/ubuntu/ \
  | grep -oP '(?<=href=")[^"]+(?=/)' \
  | grep -v '^\.\.' | grep -v '^https' | sort)
echo "Available AINIC versions in public repo:"
echo "$AVAILABLE"

if ! echo "$AVAILABLE" | grep -qx "$HOST_AINIC_VER"; then
  echo ""
  echo "ERROR: host AINIC version '$HOST_AINIC_VER' is not available in the public repo."
  echo "Available versions: $(echo $AVAILABLE | tr '\n' ' ')"
  echo "Please contact your AINIC vendor for a matching software bundle."
  exit 1
fi

echo "Found matching version '$HOST_AINIC_VER' in public repo — proceeding."

Install inside container

sudo docker exec $CONTAINER_NAME bash -c "
set -e
AINIC_VERSION=$HOST_AINIC_VER
UBUNTU_CODENAME=\$(. /etc/os-release && echo \"\$VERSION_CODENAME\")

mkdir -p /etc/apt/keyrings
curl -fsSL https://repo.radeon.com/rocm/rocm.gpg.key \
    | gpg --dearmor > /etc/apt/keyrings/amdainic.gpg
echo \"deb [arch=amd64 signed-by=/etc/apt/keyrings/amdainic.gpg] \
    https://repo.radeon.com/amdainic/pensando/ubuntu/\${AINIC_VERSION} \
    \${UBUNTU_CODENAME} main\" > /etc/apt/sources.list.d/amdainic.list

apt-get update
apt-get install -y nicctl libionic-dev ionic-common
ldconfig
ldconfig -p | grep libionic
nicctl --version
"

---

Step 3b: Install Broadcom (bnxt / thor2) userspace libraries + tools

Skip if NIC type is not thor2/bnxt.

Recommended version: for cross-node MORI (EP over RDMA / IBGDA), Broadcom firmware is solid on 237.1.137.x (official Broadcom release) and 235.2.86.x (customer-specific build). Known bad: 231.x and 232.x — if the host is on either branch, flag it to the user and recommend upgrading. Any other branch is unverified and treated the same way. The userspace library (libbnxt_re, 3b.2) must match the kernel driver version detected in 3b.1.

The bnxt path of mori check / mori setup needs two NIC-specific pieces installed here: (1) the RoCE userspace lib (libbnxt_re) and (2) a recent niccli. (dcb, the third dependency, comes with iproute2 from Step 2 — just verified in 3b.4.)

3b.1 — Detect host bnxt version (match the userspace lib to it)

Run on the host:

modinfo -F version bnxt_re 2>/dev/null || cat /sys/module/bnxt_re/version
# e.g. 235.2.88.0  → install the closest bnxt-rocelib (e.g. 235.2.86.0)

3b.2 — Install RoCE userspace lib via the Broadcom apt repo

sudo docker exec $CONTAINER_NAME bash -c '
set -e
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://packages.broadcom.com/artifactory/api/security/keypair/PackagesKey/public \
    -o /etc/apt/keyrings/broadcom-nic.asc
chmod a+r /etc/apt/keyrings/broadcom-nic.asc
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/broadcom-nic.asc] \
https://packages.broadcom.com/artifactory/ethernet-nic-debian-public jammy main" \
    > /etc/apt/sources.list.d/broadcom-nic.list
apt-get update
# pin to match the host bnxt_re version from 3b.1; list options: apt-cache madison bnxt-rocelib
apt-get install -y ibverbs-utils bnxt-rocelib=235.2.86.0
# Make the out-of-tree provider visible on a standard path. `mori check` also
# accepts the rdma-core in-tree provider (libbnxt_re-rdmav<abi>.so under a
# libibverbs dir) and honours $LD_LIBRARY_PATH, so this copy is only needed to
# have the *Broadcom* build take precedence and get its version reported.
cp /usr/local/lib/x86_64-linux-gnu/libbnxt_re* /usr/local/lib/.
ldconfig
'

Replace 235.2.86.0 with the version matching the host (3b.1). If gone from the repo, pick the nearest from apt-cache madison bnxt-rocelib.

3b.3 — Install a recent niccli (must support the qos subcommand)

mori check Step 2 uses niccli ... qos, so install one tracking the firmware (236.x/237.x for BCM57608):

sudo docker exec $CONTAINER_NAME bash -c '
set -e
NICCLI_VER=237.1.145.0          # deb version
NICCLI_PKG=237.1.148.0          # BRCM_<this> path segment in the URL
URL="https://docs.broadcom.com/docs-and-downloads/ethernet-network-adapters/NXE/BRCM_${NICCLI_PKG}/niccli/Linux/niccli-${NICCLI_VER}_linux.zip"
cd /tmp && wget -q -O niccli.zip "$URL" && unzip -o -q niccli.zip -d niccli
dpkg -i "$(find ./niccli -name "niccli_*_x86_64.deb" | head -1)"
niccli -l | head            # must list all NICs
niccli -i 1 qos --ingress --cosq --show | head   # should print the CoSQ table (TC/State/Mode)
'

Alternative (no download): host usually has a working niccli at /opt/niccli. sudo docker cp /opt/niccli $CONTAINER_NAME:/opt/ then docker exec $CONTAINER_NAME ln -sf /opt/niccli/niccli /usr/bin/niccli.

3b.4 — dcb (iproute2), needed by mori setup

sudo docker exec $CONTAINER_NAME bash -c "command -v dcb"

mori setup configures PFC/ETS via dcb and DCQCN via configfs, both at host level. With --network=host the container shares the host netns, so running it in-container is equivalent — just needs dcb + Step 1's mounts.

---

Step 3c: Install Mellanox/NVIDIA (mlx5) userspace libraries + tools

Skip if NIC type is not mlx5.