# HAMi DRA Driver
## Overview
DRA is a Kubernetes feature that lets you request and share resources among Pods, it provides a flexible way for requesting, configuring, and sharing specialized devices like GPUs. To learn more about DRA in general, good starting points are: [Kubernetes docs](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/), [Kubernetes blog](https://kubernetes.io/blog/2025/05/01/kubernetes-v1-33-dra-updates/).
The HAMi DRA Driver brings the support of DRA for [HAMi-Core](https://github.com/Project-HAMi/HAMi-core)(the in-container GPU resource controller), it might be the first open source DRA Driver for NVIDIA GPU with [DRA Consumable Capacity](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5075-dra-consumable-capacity) feature enabled.
## Installation
### Prerequisites
The list of prerequisites for running the DRA Driver for HAMi-Core is described below
- NVIDIA drivers >= 440
- nvidia-docker version > 2.0
- default runtime configured as nvidia for containerd/docker/cri-o container runtime
- `accept-nvidia-visible-devices-as-volume-mounts` option should be set to `true` for NVIDIA Container Runtime
- Kubernetes version >= 1.34 with DRA Consumable Capacity feature enabled
- glibc >= 2.17 & glibc < 2.30
- kernel version >= 3.10
Please check the [DaemonSet configuration](demo/yaml/ds.yaml) in the demo/yaml directory. We are working on a Helm Chart for this DRA Driver, it will be available in the near future.
## Demo
```yaml
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
...
spec:
devices:
requests:
- name: double-gpu-0-0
exactly:
...
capacity:
requests:
cores: 30
memory: "4Gi"
- name: double-gpu-0-1
exactly:
...
capacity:
requests:
cores: 60
memory: "8Gi"
```
Please refer to the [demo.md](demo/demo.md) for more information.
## vGPUMonitor compatibility
HAMi-Core preparation adds `hami.io/vgpu-node=<nodeName>` to the reserving Pod
so that upstream vGPUMonitor can discover it through its legacy metric selector.
This applies even when the optional monitor sidecar is disabled. The Helm chart
grants Pod `patch` permission in addition to `get/list/watch`; externally managed
RBAC must grant the same permissions before upgrading the driver.
Only that label is patched, with Pod UID and resource-version guards. An
existing matching label is accepted; a conflicting label, wrong-node Pod, or
API error fails preparation before cache edits. A Pod informer cache miss
falls back to a live API lookup, validating the reserving Pod's UID and node
before resolving its claim/container mapping. If resolution still fails,
preparation returns an error so kubelet can retry; it no longer creates a
claim-scoped `unknown` cache directory. The label is retained until Pod deletion
because other claims or containers may still need it. This is legacy-monitor
compatibility, not label-independent DRA
support. The monitor still requires NVIDIA runtime library injection.
Driver startup waits at most 30 seconds for the Pod informer's initial sync.
On timeout, startup continues with live Pod lookups while the informer keeps
retrying. Process cancellation stops the informer and aborts startup; API lookup
and labeling failures still fail preparation rather than bypassing monitoring.
Workloads previously prepared with an `unknown` cache are not migrated
automatically; recreate affected workloads through normal cleanup/preparation.
Preparation writes a `.claim-<claimUID>` ownership marker before CUDA starts;
HAMi-Core remains responsible for creating the actual `.cache` file. Unprepare
uses the marker to remove owned directories even if the workload never initializes
CUDA, the Pod has been deleted, or the driver has restarted. Explicit cleanup
also handles partially prepared claims. Older directories with the claim's
`.cache` file remain supported. Empty directories without either ownership marker
or claim cache are not deleted blindly; pre-existing unmarked orphans cannot be
attributed safely. Cache setup and ownership inspection failures are returned
as errors rather than reported as successful preparation/cleanup.
On upgrade, completed claims keep their existing CDI mounts and legacy
`/usr/local/vgpu/claims/<claimUID>/` caches. Startup, checkpoint migration and
idempotent Prepare do not sweep or relocate these directories. Unprepare removes
only the released claim's legacy directory as well as its owned new-layout
directories, even if the new `containers/` root is absent. Cleanup errors retain
the checkpoint for retry.
The safety boundary is kubelet's claim-release request: kubelet stops consuming
containers before Unprepare and defers it while another local Pod still uses
the claim. Pod API status can lag that release, so cleanup does not depend on
the Pod being absent or its phase already being terminal. Failed preparations
can also be cleaned up through the existing explicit/stale-claim Unprepare path;
there is no new legacy-directory garbage collector.
## Building the image with vGPUMonitor
`make build` and `make cmds` build this repository's Go code only. `make image`
packages the driver, HAMi-Core and upstream vGPUMonitor together. By default it
builds the monitor from HAMi's development branch, named `master` upstream:
```bash
make image
# For a repeatable build, supply the full HAMi commit SHA:
make image HAMI_VGPUMONITOR_REF=<HAMi-commit-SHA>
```
To build a development version from your HAMi fork, select its Git URL and ref:
```bash
make image HAMI_VGPUMONITOR_SOURCE=source \
HAMI_VGPUMONITOR_REPO=https://github.com/<owner>/HAMi.git \
HAMI_VGPUMONITOR_REF=<branch-or-commit-SHA>
```
`HAMI_VGPUMONITOR_REPO` defaults to `https://github.com/Project-HAMi/HAMi.git`.
The ref must exist in the selected repository and be accessible to the builder;
uncommitted changes in a local checkout are not included. The repository must
retain HAMi's module layout, including `cmd/vGPUmonitor`. The repository and ref
options apply only in source mode.
The monitor uses the target architecture's CGO cross-compiler and Go's automatic
toolchain selection to honor upstream's `go.mod`. Its source revision is logged
during the build and embedded in `vGPUmonitor version`. A moving branch is not
a reproducible dependency; pin its commit when recording test/release artifacts.
The independent `vgpumonitor-smoke` stage copies the selected binary into the
distroless runtime and executes `vGPUmonitor version`. The final image inherits
this stage and its verified binary. This does not replace GPU-runtime integration
testing.
To copy the monitor binary from HAMi v2.11.0 instead of compiling it:
```bash
make image HAMI_VGPUMONITOR_SOURCE=image \
HAMI_VGPUMONITOR_IMAGE=projecthami/hami:v2.11.0
```
Image mode requires a compatible image containing `/k8s-vgpu/bin/vGPUmonitor`
for every requested platform. HAMi v2.9.0 is not supported. Prefer a verified
image digest for reproducibility.
Both GitHub publishing workflows accept `monitor_source`, `monitor_repo`,
`monitor_ref` and `monitor_image` dispatch inputs, with corresponding repository-variable fallbacks
`HAMI_VGPUMONITOR_SOURCE`, `HAMI_VGPUMONITOR_REPO`, `HAMI_VGPUMONITOR_REF` and
`HAMI_VGPUMONITOR_IMAGE`.
All three GitHub workflows default to image mode. Their shared release-image
default is `HAMI_VGPUMONITOR_RELEASE_IMAGE` in `versions.mk`, which pins HAMi
v2.11.0's multi-platform image index for amd64 and arm64. The argument-loading
steps use this value when no image override is supplied.
Dispatch inputs override repository variables; repository variables override the
workflow fallbacks. Manual dispatch defaults `monitor_source` to `image`; select
`source` explicitly to compile a development version with `monitor_repo` and
`monitor_ref`. The source ref still defaults to `master`.
Local `make image` continues to default to source mode.
PR CI builds `vgpumonitor-smoke` for amd64 and arm64 without publishing it, using
QEMU to execute the arm64 binary's version command. This checks binary extraction
and runtime compatibility without building the driver or HAMi-Core. It also
installs Helm before running the chart tests.
## Contributing
Contributions require a Developer Certificate of Origin (DCO, see [CONTRIBUTING.md](https://github.com/Project-HAMi/HAMi/blob/master/CONTRIBUTING.md).
## Support
Please open an issue on the GitHub project for questions and for reporting problems.
Your feedback is appreciated!