Skip to main content
Enterprise

Deploy HAMi Enterprise on OpenShift

This guide is for SRE and platform teams deploying HAMi Enterprise on an OpenShift cluster with NVIDIA GPUs. It covers the OpenShift-specific security, SELinux, runtime, installation, license activation, and verification steps.

This procedure does not cover air-gapped delivery.

Prerequisites​

Before installation, confirm the following requirements:

  • The administration host has oc and Helm, and the current context points to the target cluster. License request collection also requires kubectl and jq on this host.
  • The operator has permission to create a project and cluster-scoped SecurityContextConstraints (SCC) and RBAC resources.
  • NVIDIA GPU Operator is installed, its ClusterPolicy is ready, and its built-in device plugin is disabled. HAMi Enterprise provides its own device plugin. If GPU Operator is not installed, follow NVIDIA's Installing the NVIDIA GPU Operator on OpenShift guide before continuing.
  • NVIDIA drivers and Container Toolkit are ready on every GPU node.
  • This guide uses the legacy device-plugin path. Keep dra.enabled=false; OpenShift SCC and SELinux resources are not rendered when DRA is enabled.

Inspect the cluster before continuing:

oc whoami
oc get nodes -o wide
oc get clusterpolicy
oc get nodes -L nvidia.com/gpu.present

Set the name of the GPU Operator ClusterPolicy, then verify the installation-critical fields:

export GPU_CLUSTER_POLICY='gpu-cluster-policy'

oc get clusterpolicy "$GPU_CLUSTER_POLICY" \
  -o jsonpath='state={.status.state}{"\n"}devicePlugin.enabled={.spec.devicePlugin.enabled}{"\n"}cdi.enabled={.spec.cdi.enabled}{"\n"}'

The expected output includes state=ready and devicePlugin.enabled=false. The cdi.enabled value determines which path to follow below. If either required value differs or is empty, stop and correct the GPU Operator configuration before installing HAMi Enterprise. Use the actual ClusterPolicy name if the cluster uses a different name.

Disable CDI in GPU Operator​

New GPU Operator 25.10 and later installations enable CDI by default; OLM upgrades can retain the value in the existing ClusterPolicy. This guide recommends disabling GPU Operator CDI and keeping the HAMi Enterprise defaults, devicePlugin.deviceListStrategy=envvar and scheduler.useDownward=true. Retaining the Downward API path avoids the additional per-node allocation-lock coordination used by HAMi's non-Downward path. Disabling GPU Operator CDI does not by itself improve scheduler performance.

If cdi.enabled=false already, or if the cluster must keep CDI, skip the commands in this subsection. For a cluster that keeps CDI, use the Enterprise CDI configuration later in this guide. If the GPU Operator NRI plugin is enabled, disable it before disabling CDI because NRI requires CDI; follow NVIDIA's CDI and NRI support documentation for the version-specific transition steps.

OpenShift uses CRI-O. When changing an existing ClusterPolicy, temporarily disable the GPU Operator validator on GPU nodes, disable CDI, and then restore the validator:

oc label nodes -l nvidia.com/gpu.present=true \
  nvidia.com/gpu.deploy.operator-validator=false --overwrite

oc patch clusterpolicy "$GPU_CLUSTER_POLICY" --type='json' \
  -p='[{"op":"replace","path":"/spec/cdi/enabled","value":false}]'

oc label nodes -l nvidia.com/gpu.present=true \
  nvidia.com/gpu.deploy.operator-validator=true --overwrite

If the patch command fails, still run the final label command to restore the validator before troubleshooting the failure.

Wait until the ClusterPolicy is ready again, then confirm the result:

oc get clusterpolicy "$GPU_CLUSTER_POLICY" \
  -o jsonpath='state={.status.state}{"\n"}cdi.enabled={.spec.cdi.enabled}{"\n"}'

The expected output is state=ready and cdi.enabled=false. For the GPU Operator ClusterPolicy, use spec.cdi.enabled; setting the deprecated GPU Operator Helm value cdi.default=false does not disable CDI.

Set the Chart version​

Set the HAMi Enterprise Chart and version:

export HAMI_ENTERPRISE_CHART='oci://ghcr.io/dynamia-ai/charts/hami-enterprise'
export HAMI_ENTERPRISE_VERSION='REPLACE_WITH_VERSION'

Prepare the OpenShift project​

Create a dedicated project for HAMi Enterprise. Do not install it in a highly privileged platform project such as kube-system.

oc new-project hami-system

By default, the HAMi device-plugin runs only on nodes labeled gpu=on. If you use the default selector, set GPU_NODE and label every NVIDIA node that HAMi should manage:

export GPU_NODE='REPLACE_WITH_GPU_NODE_NAME'
oc label node "$GPU_NODE" gpu=on --overwrite

To use the existing nvidia.com/gpu.present label instead, skip the labeling command and follow the alternative configuration in the next section.

Configure HAMi Enterprise​

Keep the default HAMi Enterprise envvar injection and Downward API settings. Because scheduler.useDownward already defaults to true, it is not repeated in the values file. Set devicePlugin.runtimeClassName=nvidia so that the HAMi device plugin uses the NVIDIA runtime even when it is not the cluster default.

Create values-openshift.yaml. The file contains only values that differ from the Chart defaults:

platform:
  openshift: true

selinux:
  enabled: true

scheduler:
  service:
    type: ClusterIP

devicePlugin:
  runtimeClassName: nvidia

When GPU Operator manages the NVIDIA driver, the driver root on the node is /run/nvidia/driver, with NVML libraries under /run/nvidia/driver/usr/lib64/, rather than under /. Selecting the nvidia RuntimeClass makes NVIDIA Container Toolkit expose these files inside containers at startup. The recommended envvar configuration does not need devicePlugin.nvidiaDriverRoot. If the HAMi device plugin or a HAMi-scheduled workload fails to start, follow NVIDIA containers fail with GPU Operator 25.10+.

Configure devicePlugin.nvidiaDriverRoot and devicePlugin.nvidiaHookPath according to how the driver and NVIDIA Container Toolkit are installed on the GPU nodes. The following descriptions include common installation paths; use the paths that match the actual deployment:

  • nvidiaDriverRoot is the NVIDIA driver installation root on the host. GPU Operator-managed drivers commonly use /run/nvidia/driver; drivers installed directly on the host commonly use /. Set it to the actual driver root, not the directory containing an individual NVML library.
  • nvidiaHookPath is the host executable path used for the NVIDIA CDI hook in the generated CDI specification. /usr/local/nvidia/toolkit/nvidia-ctk applies only when Toolkit installs the executable there; use the actual executable path for a preinstalled or customized Toolkit.

Both Chart values default to empty. Decide whether to override each one from the node's installation layout and the device plugin's effective defaults; do not copy the two example paths solely because CDI is enabled.

For other HAMi Enterprise Chart values, defaults, and parameter descriptions, see the HAMi Enterprise Chart values reference. This reference is generated from the Chart by helm-docs.

Use the NVIDIA GPU node label​

If you did not add gpu=on, confirm that nvidia.com/gpu.present=true identifies every NVIDIA node that HAMi should manage. Set GPU_NODE to one of those nodes for the later verification commands:

oc get nodes -l nvidia.com/gpu.present=true
export GPU_NODE='REPLACE_WITH_GPU_NODE_NAME'

If you use this label, replace the devicePlugin block in the preceding example with:

devicePlugin:
  runtimeClassName: nvidia
  nvidiaNodeSelector:
    gpu: null
    nvidia.com/gpu.present: "true"

The gpu: null entry removes the Chart-default selector. Without it, Helm merges both map entries and the device-plugin still requires gpu=on.

Use CDI only when required​

If the environment requires CDI, keep GPU Operator spec.cdi.enabled=true. In the existing values-openshift.yaml, preserve the selected node selector and other OpenShift settings, and apply the following HAMi Enterprise overrides. HAMi CDI requires scheduler.useDownward=false; the example assumes that the GPU Operator NRI plugin is disabled and that the nvidia RuntimeClass exists.

scheduler:
  useDownward: false

devicePlugin:
  runtimeClassName: nvidia
  deviceListStrategy: cdi-annotations
  # Example paths; adjust each to the node installation layout.
  nvidiaDriverRoot: /run/nvidia/driver
  nvidiaHookPath: /usr/local/nvidia/toolkit/nvidia-ctk

The open-source HAMi CDI guide provides CDI background and verification steps, but its Helm values do not include the HAMi Enterprise scheduler.useDownward=false requirement. Use the Enterprise override above for this OpenShift installation.

Configure OpenShift security​

With platform.openshift enabled, the Chart defaults create a dedicated hami-device-plugin SCC, its system:openshift:scc:hami-device-plugin ClusterRole, and a RoleBinding for the enabled device-plugin ServiceAccount. The SCC permits privileged containers, host PID, hostPath volumes, and the SYS_ADMIN capability, while host IPC, host networking, and host ports remain disabled. The scheduler and admission ServiceAccounts are not included in this grant and continue to use an SCC selected by OpenShift.

Review the dedicated SCC against the cluster security policy before installation. To use an existing SCC instead, set create: false and set name to privileged or another approved SCC whose corresponding system:openshift:scc:<name> ClusterRole exists. To manage the grant entirely outside the Chart, set create: false, leave name empty, and grant the approved SCC only to the rendered device-plugin ServiceAccount.

SELinux relabeling uses the Chart-default container_file_t type and s0 level. Enabling it changes labels and directory modes on these host paths:

/usr/local/vgpu
/usr/local/vgpu/containers
/tmp/vgpulock

Install HAMi Enterprise​

Install HAMi Enterprise:

helm upgrade --install hami "$HAMI_ENTERPRISE_CHART" \
  --version "$HAMI_ENTERPRISE_VERSION" \
  --namespace hami-system \
  --create-namespace \
  -f values-openshift.yaml \
  --wait

Verify the deployment​

Check the scheduler and device-plugin rollout:

oc rollout status deployment/hami-hami-enterprise-scheduler -n hami-system
oc rollout status daemonset/hami-hami-enterprise-device-plugin -n hami-system
oc get pods -n hami-system -o wide

Check the SCC assigned to each running Pod:

oc get pods -n hami-system \
  -o 'custom-columns=NAME:.metadata.name,SCC:.metadata.annotations.openshift\.io/scc'

The device-plugin Pods should use the SCC selected for the configured grant mode: hami-device-plugin with the Chart defaults, or the approved existing SCC for the other modes. Scheduler Pods should use an SCC selected by OpenShift, normally a restricted SCC that is valid for the project. Successful admission hook Jobs use the Helm hook-succeeded deletion policy, so their Pods are normally absent at this stage; inspect them only when a failed hook leaves them behind.

Confirm the GPU Operator state:

oc get clusterpolicy
oc get node "$GPU_NODE" -L gpu,nvidia.com/gpu.present
oc describe node "$GPU_NODE"

Check the HAMi host directory SELinux labels on a GPU node:

oc debug node/"$GPU_NODE" -- chroot /host \
  ls -Zd /usr/local/vgpu /usr/local/vgpu/containers /tmp/vgpulock

The three HAMi paths should show the configured container_file_t type at the s0 level.

Obtain a license​

Wait until the installed HAMi Enterprise Pods have started before collecting the license request information. The collection script uses kubectl; confirm that kubectl uses the same target cluster as oc, and that jq is available:

kubectl config current-context
jq --version

Download and run the collection script:

curl -fsSLO https://public.hami.run/collect-hami-license-info.sh
bash collect-hami-license-info.sh

If the delivery package already contains the script, run the local copy instead:

bash collect-hami-license-info.sh

The script outputs license request information in the following format:

{
  "esn": "96565d61-986a-4918-aafb-448ff6e3746b",
  "deviceInstances": [
    {
      "uuid": "GPU-ceee905d-48ac-93de-a81b-17c00e1e5e02",
      "deviceType": "NVIDIA A10"
    }
  ]
}

Send the complete JSON output to Dynamia.ai through the dedicated support channel for your commercial agreement. You can also contact info@dynamia.ai or call 400-026-7800 to obtain the license.

Activate the license​

After receiving the license file, save it on the administration host, then create and label the license Secret:

kubectl create secret generic hami-license \
  --from-file=license=/path/to/license-file \
  -n hami-system

kubectl label secret hami-license \
  hami.io/license="true" \
  -n hami-system

Verify the Secret and the license validation event:

kubectl get secret hami-license -n hami-system
kubectl get events --field-selector involvedObject.name=hami-license -n hami-system
kubectl get nodes \
  -o custom-columns='NODE:.metadata.name,LICENSE:.metadata.annotations.hami\.io/nvidia-license'

A LicenseValid event indicates that license validation succeeded. For NVIDIA nodes, the LICENSE column also shows the license registration annotation. Complete activation before testing vGPU partitioning and scheduling, then submit the workload used by your acceptance procedure.

Troubleshooting​

SymptomChecks and resolution
Rendered manifests do not contain OpenShift resources or selinux-relabelConfirm that values-openshift.yaml enables both platform.openshift and selinux.enabled, then render the manifests again.
Device-plugin Pods are rejected by SCC admissionCheck the SCC selected for the configured grant mode and the SCC annotation on the device-plugin Pods. For a Chart-managed grant, inspect the rendered RoleBinding and device-plugin ServiceAccount. For an externally managed grant, inspect the external binding. Do not grant the SCC to scheduler, admission, or application ServiceAccounts.
Expected SCC-mode resources or SELinux resources are absentConfirm dra.enabled=false, then compare the render with the selected SCC grant mode. A dedicated SCC is intentionally absent when create: false; all legacy SCC and SELinux resources are intentionally absent when DRA is enabled.
Device-plugin Pods remain PendingConfirm the node matches devicePlugin.nvidiaNodeSelector, inspect node taints, and verify that devicePlugin.tolerations matches them.
Device-plugin starts but reports NVIDIA hook or libcuda.so.1 errorsCheck the NVIDIA drivers, Container Toolkit, and GPU Operator state. This indicates a runtime integration problem, not proof of a HAMi scheduling failure. Follow NVIDIA containers fail with GPU Operator 25.10+, and see Is HAMi compatible with NVIDIA GPU Operator and DCGM metrics? for compatibility background.
A workload cannot access HAMi shared directoriesCheck the selinux-relabel logs and the host labels on the HAMi shared directories.

Uninstalling the Helm release does not restore SELinux labels or directory modes changed on the host. If the nodes must return to their previous state, record the original state before installation and restore it according to the organization's node maintenance procedure.