k8s集群节点配置GPU
默认已有以下前置条件
- k8s 1.10+
- 操作系统:麒麟v10 SP2(Centos,Ubuntu等见“安装NVIDIA Container Toolkit”中其他源配置)
- GPU驱动:已安装
官方文档:https://github.com/NVIDIA/k8s-device-plugin
一、验证GPU挂载以及驱动是否安装
验证是否挂载
$ lspci -vv | grep -i nvidia
00:0d.0 VGA compatible controller: NVIDIA Corporation TU104GL [Quadro RTX 5000] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation Device 129f
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_drm, nvidia
00:0e.0 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)
Subsystem: NVIDIA Corporation Device 129f
00:0f.0 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1) (prog-if 30 [XHCI])
Subsystem: NVIDIA Corporation Device 129f
00:10.0 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)
Subsystem: NVIDIA Corporation Device 129f
验证驱动是否安装
nvidia-smi
全称是NVIDIA System Management Interface ,它是一个基于前面介绍过的NVIDIA Management Library(NVML)
构建的命令行实用工具,旨在帮助管理和监控NVIDIA GPU设备。
# nvidia-smi
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.113.01 Driver Version: 535.113.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Quadro RTX 5000 Off | 00000000:00:0D.0 Off | Off |
| 29% 23C P8 12W / 230W | 0MiB / 17236MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
查看动态监控显卡状态
$ watch -t -n 1 nvidia-smi
验证cuda版本
nvcc -V
二、 增加GPU节点标签
$ kubectl label nodes 节点名称 node-role.kubernetes.io/gpu=true
三、在容器中启用 GPU 支持
需要安装NVIDIA Container来支持docker调用显卡
如不设置则会报如下错误:
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].
NVIDIA Container主要组件包括nvidia-container-runtime
, nvidia-container-toolkit
, libnvidia-container
,
nvidia-container-toolkit:CUDA Toolkit是一个用于开发和优化CUDA应用程序的软件包,其中包括CUDA驱动程序和CUDA运行时库。
nvidia-container-runtime:CUDA运行时库是一个用于在GPU上执行CUDA应用程序的软件组件,它提供了一组CUDA API函数,于管理GPU内存和执行CUDA内核。
官方架构地址:https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/arch-overview.html
依赖关系:
├─ nvidia-container-toolkit (version)
│ ├─ libnvidia-container-tools (>= version)
│ └─ nvidia-container-toolkit-base (version)
│
├─ libnvidia-container-tools (version)
│ └─ libnvidia-container1 (>= version)
└─ libnvidia-container1 (version)
由于nvidia-container-toolkit已包含运行时,这里我们选择安装nvidia-container-toolkit,用于后续k8s节点使用GPU
安装NVIDIA Container Toolkit
运行NVIDIA Container Toolkit的条件:
内核版本 > 3.10 的 GNU/Linux x86_64
Docker >= 19.03(推荐,但某些发行版可能包含旧版本的 Docker。支持的最低版本为 1.12)
架构 >= Kepler(或计算能力 3.0)的 NVIDIA GPU
NVIDIA Linux 驱动程序>= 418.81.07(请注意,不支持较旧的驱动程序版本或分支。)
- 麒麟系统 v10 sp2设置源:
$ curl -s -L https://nvidia.github.io/nvidia-docker/centos8/nvidia-docker.repo | sudo tee /etc/yum.repos.d/nvidia-docker.repo
$ sudo yum install -y nvidia-container-toolkit
- CentOS源
$ curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
$ sudo yum install -y nvidia-container-toolkit
- Ubuntu源
$ curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list \
&& \
sudo apt-get update
$ sudo apt-get install -y nvidia-container-toolkit
其他系统存储库配置:https://nvidia.github.io/nvidia-docker/
Docker显卡适配
在Docker在19.03版本以后直接安装NVIDIA Container,可以不需要再独立安装nvidia-docker也就是docker2直接设置就可以支持显卡
直接设置daemon.json,增加如下
"default-runtime": "nvidia",
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
}
配置完成后进行重启
$ sudo systemctl restart docker
Cotainerd显卡适配
在config.toml,增加如下:
version = 2
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "nvidia"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
privileged_without_host_devices = false
runtime_engine = ""
runtime_root = ""
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
BinaryName = "/usr/bin/nvidia-container-runtime"
配置完成后进行重启
sudo systemctl restart containerd
容器内验证
docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
Tue Nov 14 12:29:39 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.113.01 Driver Version: 535.113.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Quadro RTX 5000 Off | 00000000:00:0D.0 Off | Off |
| 29% 23C P8 12W / 230W | 0MiB / 17236MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
如上为成功设置
四、在Kubernetes中启用 GPU 支持
安装k8s插件 nvidia-device-plugin
$ kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.2/nvidia-device-plugin.yml
nvidia-device-plugin.yml内容:
# Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nvidia-device-plugin-daemonset
namespace: kube-system
spec:
selector:
matchLabels:
name: nvidia-device-plugin-ds
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
name: nvidia-device-plugin-ds
spec:
tolerations:
- key: node-usage.project/name
operator: Equal
value: gpu
priorityClassName: system-node-critical
containers:
- image: 'nvidia/k8s-device-plugin:1.11'
name: nvidia-device-plugin-ctr
env:
- name: FAIL_ON_INIT_ERROR
value: "false"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
volumeMounts:
- name: device-plugin
mountPath: /var/lib/kubelet/device-plugins
volumes:
- name: device-plugin
hostPath:
path: /var/lib/kubelet/device-plugins
运行后查看打标签的节点gpu是否启用
$ kubectl get nodes "-o=custom columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
NAME GPU
k8s-worker01 <none>
k8s-worker02 1
k8s-worker03 1
k8s部署验证
部署job进行验证
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: gpu-pod
spec:
restartPolicy: Never
containers:
- name: cuda-container
image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
resources:
limits:
nvidia.com/gpu: 1 # requesting 1 GPU
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
EOF
如下输出则k8s启用GPU成功
$ kubectl logs gpu-pod
Loading NVML
Fetching devices.
Starting FS watcher.
Starting OS watcher.
Starting to serve on /var/lib/kubelet/device-plugins/nvidia.sock
Registered device plugin with Kubelet