T-Head PPU
PPU Device Plugin registers T-Head PPUs, such as Zhenwu 810E, as Kubernetes resources for memory and compute sharing. The HAMi scheduler selects devices; Pods that bypass it fail allocation. This guide uses Chart 0.0.4 and application v1.0.5 to configure licensing, install the plugin, and verify a workload.
Prerequisites
- Kubernetes 1.20 or later, and Helm 3.
- PPU drivers are installed on each device node, which exposes
/dev/alixpu,/dev/alixpu_ctl, and/dev/alixpu_ppu{N}. - The HAMi Enterprise scheduler is deployed, and its resource names match this plugin. See Installation for deployment options.
- An Enterprise license covers the target PPU devices.
Install
kubectl label node <node-name> alibabappu=on
helm upgrade --install ppu-vgpu-device-plugin \
oci://ghcr.io/dynamia-ai/charts/ppu-vgpu-device-plugin \
--version 0.0.4 \
-n hami-system --create-namespace
Chart 0.0.4 uses image ghcr.io/dynamia-ai/ppu-device-plugin:v1.0.5. To pull the image from another registry, set image.repository and image.tag.
Verify the install:
kubectl -n hami-system get ds -l app.kubernetes.io/instance=ppu-vgpu-device-plugin
kubectl get node <node-name> -o jsonpath='{.status.allocatable.alibaba\.com/ppu}{"\n"}'
allocatable equals the physical card count multiplied by deviceSplitCount.
Uninstall:
helm uninstall ppu-vgpu-device-plugin -n hami-system
Workload resources
| Resource | Description |
|---|---|
alibaba.com/ppu | Number of cards |
alibaba.com/ppu-memory | Memory per card, in MiB |
alibaba.com/ppu-memory-percentage | Memory percentage per card (1–100) |
alibaba.com/gpu-core | Compute percentage per card (1–100) |
Requesting only the card count reserves the whole card. ppu-memory takes precedence over ppu-memory-percentage.
Configuration
The following values belong to the plugin chart under config.data.alibaba:
| Parameter | Default | Description |
|---|---|---|
deviceSplitCount | 10 | Maximum number of containers that share one card |
deviceMemoryScaling | 1.0 | Memory oversubscription factor |
deviceCoreScaling | 1.0 | Compute oversubscription factor |
resourceCountName and related names | alibaba.com/* | Resource names; keep them identical to the scheduler |
nodeSelector defaults to alibabappu: "on" and selects the nodes that run the plugin. monitor.service.nodePort defaults to 31992.
Pod annotations:
| Annotation | Description |
|---|---|
alibaba.com/use-pputype / nouse-pputype | Select or exclude a model |
alibaba.com/use-ppuuuid / nouse-ppuuuid | Select or exclude a UUID |
hami.io/gpu-scheduler-policy | binpack, spread, or topology-aware |
Smoke test
Run two vLLM instances on the same PPU. Each instance requests 50% memory and 50% compute. The test checks sharing, limit injection, and inference.
This guide does not provide a test image. Prepare an image that includes the PPU SDK, PyTorch, and vLLM, and make sure the cluster can pull it.
export PPU_IMAGE=<ppu-vllm-image>
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: ppu-smoke
spec:
replicas: 2
selector:
matchLabels:
app: ppu-smoke
template:
metadata:
labels:
app: ppu-smoke
annotations:
hami.io/node-scheduler-policy: binpack
hami.io/gpu-scheduler-policy: binpack
spec:
containers:
- name: vllm
image: ${PPU_IMAGE}
command:
- sh
- -c
- |
pip install modelscope &&
modelscope download --model Qwen/Qwen2.5-1.5B-Instruct --local_dir /models/qwen &&
exec python3 -m vllm.entrypoints.openai.api_server \
--model /models/qwen --served-model-name qwen \
--trust-remote-code --gpu-memory-utilization 0.8
readinessProbe:
httpGet:
path: /health
port: 8000
periodSeconds: 10
failureThreshold: 90
resources:
limits:
cpu: "8"
memory: 16Gi
alibaba.com/ppu: 1
alibaba.com/ppu-memory-percentage: 50
alibaba.com/gpu-core: 50
volumeMounts:
- name: models
mountPath: /models
volumes:
- name: models
emptyDir: {}
EOF
kubectl rollout status deployment/ppu-smoke --timeout=30m
Check the result:
for p in $(kubectl get pod -l app=ppu-smoke -o name); do
echo "== $p"
kubectl get $p -o jsonpath='{.metadata.annotations.hami\.io/vppu-devices-allocated}{"\n"}'
kubectl exec $p -- sh -c 'env | grep CUDA_DEVICE_'
kubectl exec $p -- python3 -c 'import torch; print("total", torch.cuda.mem_get_info()[1] >> 20, "MiB")'
kubectl exec $p -- curl -s localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "qwen", "messages": [{"role": "user", "content": "Hello"}]}'
echo
done
Pass conditions:
| Check | Expected |
|---|---|
| Allocation annotation | Both Pods use the same device UUID |
CUDA_DEVICE_MEMORY_LIMIT_0 | About 50% of the full card memory |
CUDA_DEVICE_SM_LIMIT | 50 |
total | Matches CUDA_DEVICE_MEMORY_LIMIT_0 |
| Inference response | choices[0].message.content is non-empty |
Clean up:
kubectl delete deployment ppu-smoke
Troubleshooting
kubectl -n hami-system logs -l app.kubernetes.io/component=device-plugin --tail=200
| Symptom | Check |
|---|---|
| No plugin Pod | Node label alibabappu=on |
No alibaba.com/ppu resource | Driver, /dev/alixpu*, and plugin logs |
device request not found | The Pod schedulerName |
| License error | Device UUIDs and expiry in node annotation hami.io/alibaba-license, license coverage for HGML-visible devices, workload timezone, and whether workload processes were restarted after renewal |