kata-containers

name

kata-containers

description

Expert guide for Kata Containers: GPU passthrough (VFIO/IOMMU), microVM architecture, installation, troubleshooting, K8s integration, and the NVIDIA 32-GPU-per-driver scaling solution. Use this skill whenever the user mentions kata containers, kata-runtime, kata-qemu shim, VFIO passthrough, GPU passthrough to containers, IOMMU groups, microVM containers, the NVIDIA NV_MAX_DEVICES 32-GPU limit, GPU Operator sandbox mode, CDI with VMs, or container workloads needing kernel-level isolation. Also trigger for questions like "how do I pass a GPU into a container with strong isolation" or "can containers bypass the NVIDIA 32-GPU driver limit."

Kata Containers Skill

Kata Containers runs each pod/container inside a lightweight microVM (via QEMU/Cloud Hypervisor). The container user experience is identical to runc, but each workload gets its own guest kernel — enabling full device passthrough (VFIO) and true kernel isolation.

Why This Matters: The NVIDIA 32-GPU Ceiling

NVIDIA's driver hard-codes NV_MAX_DEVICES = 32 — one nvidia.ko instance can manage at most 32 GPUs. On a 64-GPU host, the remaining 32 are simply invisible.

The only fix: run multiple independent driver instances, each inside its own kernel — i.e., microVMs.

• Host: zero NVIDIA drivers, only vfio-pci holding GPUs
• VM 1: nvidia.ko manages GPU 0-31
• VM 2: nvidia.ko manages GPU 32-63

Kata Containers (not runc/Docker) makes this transparent to K8s workloads.

Architecture: 4-Layer Stack

[Host kernel] — only vfio-pci, no nvidia.ko
      ↓  VFIO / IOMMU passthrough
[QEMU microVM] — guest kernel + nvidia.ko
      ↓  vsock ttrpc
[kata-agent] — receives OCI spec, manages container inside VM
      ↓
[Container process] — sees /dev/nvidia0, runs CUDA normally

Four core components:

• containerd-shim-kata-v2: Host-side OCI runtime shim
• QEMU / Cloud Hypervisor: Launches the microVM
• Guest kernel + rootfs (kata-containers.img): Minimal Linux inside VM
• kata-agent: Daemon inside guest, executes container ops

PCIe / VFIO / IOMMU Concepts

BDF — PCIe device address: 0000:01:00.0 (domain:bus:device.function)

IOMMU — Hardware unit that translates DMA addresses. Without it, a GPU inside a VM could DMA-read all host memory. Prerequisite for passthrough.

IOMMU Group — Devices sharing DMA translation. Must pass through as a complete group.

vfio-pci — Kernel module that claims a device from its normal driver, exposing /dev/vfio/<group_id>.

Key rule for multi-function GPUs (e.g. RTX 3080):

• Bind both functions to vfio-pci → Triggers Kata issue #12549 (QEMU hang)
• Bind only VGA to vfio-pci, leave audio → Illegal — group must be whole
• Bind only VGA to vfio-pci, remove audio from PCI tree → Correct

echo 1 > /sys/bus/pci/devices/0000:01:00.1/remove

Driver Layer Alignment

Three layers must be in sync:

Host kernel     → MUST have vfio-pci, MUST NOT have nvidia.ko
Guest kernel    → MUST have nvidia.ko matching the guest kernel version (NOT host kernel!)
Container image → CUDA userspace libraries and nvidia-smi (from nvcr.io/nvidia/cuda)

Using kata-qemu-nvidia-gpu shim (not plain kata-qemu) brings the pre-installed nvidia.ko guest image.

Quick Installation Path (Proven: kata-static 3.29.0 + RTX 3080)

See references/v4-implementation.md for the full step-by-step guide.

Summary of 5 critical steps:

1. BIOS: Enable VT-d/AMD-Vi, Above 4G Decoding, disable Secure Boot
2. Host: Blacklist nvidia + snd_hda_intel; bind only VGA function to vfio-pci; remove audio function
3. Install: kata-static-3.29.0 tarball to /opt/kata/
4. Configure: Copy configuration-qemu-nvidia-gpu.toml, set pod_resource_api_sock="" and default_memory=4096
5. vsock: modprobe vsock vhost_vsock vmw_vsock_virtio_transport

Final verification:

GID=$(basename "$(readlink /sys/bus/pci/devices/0000:01:00.0/iommu_group)")
ctr -n default run --rm \
  --runtime io.containerd.kata-qemu-nvidia-gpu-ctr.v2 \
  --device /dev/vfio/${GID} \
  docker.io/nvidia/cuda:12.4.1-base-ubuntu22.04 \
  test-$(date +%H%M%S) nvidia-smi

Layer-by-Layer Verification Checklist

Layer	Command	Expected Result
BIOS/IOMMU	dmesg	grep -E 'DMAR
vfio-pci holding GPU	lspci -nnk -d 10de:2206	Kernel driver in use: vfio-pci
No nvidia on host	lsmod	grep nvidia
vsock available	ls /dev/vhost-vsock	file exists
Plain Kata works	ctr run --runtime io.containerd.kata.v2 --rm alpine:latest t uname -r	different kernel from host
GPU visible	nvidia-smi inside container	GPU table printed

Always verify each layer before moving to the next.

Top Failure Modes

Symptom	Root Cause	Fix
kata-runtime check fails IOMMU	grub param not applied	Check /proc/cmdline, update-grub reboot
Kernel driver in use: nvidia (not vfio-pci)	nvidia driver on host	apt purge nvidia, rebuild initramfs, reboot
QEMU hangs (vsock timeout)	Multi-function device #12549 or DMA pinning	Remove audio function; enable hugepages+prealloc
No devices were found	Wrong shim — no nvidia.ko in guest	Use kata-qemu-nvidia-gpu shim
Driver/library version mismatch	CUDA newer than guest driver	Lower CUDA image version
host system doesn't support vsock	vhost_vsock module missing	modprobe vsock vhost_vsock vmw_vsock_virtio_transport
containerd-shim-kata-v2: file does not exist	/opt/kata/bin not in containerd PATH	Add systemd drop-in for PATH
pod_resource_api_sock kubelet error	Running bare ctr without K8s	Set pod_resource_api_sock="" in config
QEMU startup hang (large memory)	DMA pinning 4KB pages for 32GB	enable_hugepages=true + enable_mem_prealloc=true
BAR allocation failure	BIOS Above 4G Decoding off	Enable in BIOS

When QEMU hangs, do NOT kill it. Run this first:

QPID=$(pgrep -n qemu)
sudo cat /proc/$QPID/stack
sudo cat /proc/$QPID/status | grep VmLck
sudo cat /run/vc/sbs/*/qemu.log 2>/dev/null | tail -100

Stack signatures:

• vfio_pin_pages_remote → DMA pinning → enable hugepages
• pci_bridge_update_mappings → PCIe topology → check multi-function
• schedule_timeout idle → agent timeout → raise KATA_AGENT_TIMEOUT

K8s Integration

See references/k8s-integration.md for: RuntimeClass, GPU Operator sandbox mode helm values, CDI spec, Device Plugin, full pod YAML.

Quick RuntimeClass:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata-qemu-nvidia-gpu
handler: kata-qemu-nvidia-gpu

Pod using it:

spec:
  runtimeClassName: kata-qemu-nvidia-gpu
  containers:
  - resources:
      limits:
        nvidia.com/pgpu: "1"

Scaling to 64+ GPUs

Host (no nvidia.ko, vfio-pci holds all 64 GPUs)
├── Kata VM 1: nvidia.ko → GPU 0-31  (legal: ≤32 per instance)
└── Kata VM 2: nvidia.ko → GPU 32-63 (legal: ≤32 per instance)

Config for 32-GPU VMs:

• pcie_root_port = 64 (32 GPUs x 2 functions)
• Use 1GB hugepages: hugepagesz=1G (not 2MB — too slow at scale)
• QEMU >= 8.2 (older versions have multi-VFIO concurrency bugs)
• NUMA alignment: match VM vCPUs to same NUMA node as GPUs

Exception: HGX/NVSwitch full-mesh machines — NVSwitch fabric manager needs all GPUs visible. Splitting breaks NCCL all-reduce. Don't split these.

CTK vs CDI vs Device Plugin

• CTK (nvidia-container-toolkit): runc + host driver only. Useless for Kata — driver is in guest.
• Device Plugin: K8s resource scheduling. Registers nvidia.com/pgpu with kubelet.
• CDI: Declarative device injection. Maps pgpu index to /dev/vfio/. Works with Kata.

For Kata GPU passthrough in K8s: Device Plugin (scheduling) + CDI (device injection). Never CTK.

Common Mistakes to Avoid

1. Don't install nvidia-container-toolkit on host in Kata GPU mode — it's for runc only
2. Don't build guest rootfs manually — use official kata-static NVIDIA guest images
3. Don't start with hugepages/prealloc in first validation — use 4GB guest to isolate GPU issues
4. Don't upgrade Kata version mid-debug — config key semantics differ between versions
5. Don't bind both GPU functions to vfio-pci — remove the audio function instead
6. Don't compile the shim with go1.24+ if on Kata 3.13.x — use go1.22

Reference Files

• references/v4-implementation.md — Complete step-by-step guide (proven: Ubuntu 24.04 + RTX 3080 + Kata 3.29.0)
• references/k8s-integration.md — K8s RuntimeClass, GPU Operator sandbox, CDI, pod YAML