Ubuntu 22.04 K8S v1.24.6 containerd 配置共享GPU集群调度

背景

全球主要的容器集群服务厂商的 Kubernetes 服务都提供了 Nvidia GPU 容器调度能力，但是通常都是将一个GPU卡分配给一个容器。这可以实现比较好的隔离性，确保使用GPU的应用不会被其他应用影响；对于深度学习模型训练的场景非常适合，但是如果对于模型开发和模型预测的场景就会比较浪费。大家的诉求是能够让更多的预测服务共享同一个GPU卡上，进而提高集群中 Nvidia GPU的利用率。
所以在这篇文章的基础上 https://www.cnblogs.com/klvchen/articles/17622898.html 进行修改

注意：下面步骤都在 master 节点上操作

部署GPU共享调度程序扩展程序

mkdir -p /data/yaml/kube-system/gpushare-scheduler-extender/
cd /data/yaml/kube-system/gpushare-scheduler-extender/
wget https://raw.githubusercontent.com/AliyunContainerService/gpushare-scheduler-extender/master/config/gpushare-schd-extender.yaml

# 这里需要部署在 master 节点上,需要对 gpushare-schd-extender.yaml 进行修改
vi gpushare-schd-extender.yaml
# 把 
node-role.kubernetes.io/master: ""
# 改成
node-role.kubernetes.io/control-plane: ""

# 执行，并检查容器是否正常启动
kubectl apply -f gpushare-schd-extender.yaml
kubectl -n kube-system get pod

修改 scheduler 的配置

cd /etc/kubernetes
curl -O https://raw.githubusercontent.com/AliyunContainerService/gpushare-scheduler-extender/master/config/scheduler-policy-config.yaml

cd /tmp
wget https://raw.githubusercontent.com/AliyunContainerService/gpushare-scheduler-extender/master/config/kube-scheduler-v1.23%2B.yaml
mv kube-scheduler-v1.23+.yaml kube-scheduler.yaml

vi kube-scheduler.yaml
# 这里需要把 image 改成 image: registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.6
# 注意， static pod 的配置在 /etc/kubernetes/manifest，修改时要覆盖，不能直接操作
cd /etc/kubernetes/manifests/
cp /tmp/kube-scheduler.yaml .

# 查容器是否正常启动
kubectl -n kube-system get pod

部署设备插件

cd /data/yaml/kube-system/gpushare-scheduler-extender/
wget https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-rbac.yaml
wget https://raw.githubusercontent.com/AliyunContainerService/gpushare-device-plugin/master/device-plugin-ds.yaml

# 事先删除原来的 GPU device plugin
helm -n nvidia-device-plugin delete nvdp

# 注意，只有打了标签的节点 gpushare-device-plugin 容器才会启动
kubectl apply -f device-plugin-rbac.yaml
kubectl apply -f device-plugin-ds.yaml

# 添加 gpushare 节点标签
kubectl label node alg-k8s-master-01 gpushare=true

# 查容器是否正常启动
kubectl -n kube-system get pod

下载并安装 kubectl 扩展

mkdir -p  /data/software/
cd /data/software/
wget https://github.com/AliyunContainerService/gpushare-device-plugin/releases/download/v0.3.0/kubectl-inspect-gpushare
cp kubectl-inspect-gpushare /usr/bin/
chmod u+x /usr/bin/kubectl-inspect-gpushare

# 使用实例查询共享图形处理器的分配状态
kubectl inspect gpushare
kubectl inspect gpushare -d

测试

mkdir -p /data/yaml/default/test && cd /data/yaml/default/test

cat >> test.yaml << EOF
apiVersion: apps/v1 
kind: StatefulSet

metadata:
  name: binpack-1
  labels:
    app: binpack-1

spec:
  replicas: 3
  serviceName: "binpack-1"
  podManagementPolicy: "Parallel"
  selector: # define how the deployment finds the pods it manages
    matchLabels:
      app: binpack-1

  template: # define the pods specifications
    metadata:
      labels:
        app: binpack-1

    spec:
      containers:
      - name: binpack-1
        image: cheyang/gpu-player:v2
        resources:
          limits:
            # GiB
            aliyun.com/gpu-mem: 3
EOF

kubectl apply -f test.yaml

# 检查
kubectl get pod
kubectl inspect gpushare