【异常修复】Docker启动容器时无法使用GPU

硬件配置

显卡：Tesla V100

问题呈现

1. 某次重启后，发现服务器nvidia-docker无法start已有的部分容器

$ docker start ******
Error response from daemon: failed to create shim: OCI runtime create failed
: container_linux.go:380: starting container process caused
: process_linux.go:545: container init caused
: Running hook #0:: error running hook: exit status 1, stdout: , stderr
: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown
Error: failed to start containers: ******

2. 为排查问题的可能原因，尝试另行启动容器，发现启动时无法使用GPU

$ docker run -it --gpus all [Image_Name]:[Image_Tag]
docker: 
Error response from daemon: failed to create shim: OCI runtime create failed
: container_linux.go:380: starting container process caused
: process_linux.go:545: container init caused
: Running hook #0:: error running hook: exit status 1, stdout: , stderr
: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown.

3. 为了进一步排查问题的可能原因，使用nvidia-smi命令查看显卡情况，发现异常

$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. 
Make sure that the latest NVIDIA driver is installed and running.

问题定位

docker报错中包含nvml error: driver not loaded: unknown，因此问题产生原因可能是显卡驱动被挤掉了

解决方案

1. 查验现状
   用nvcc -V确认显卡驱动存在

$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0

2. 查看已安装驱动的版本信息

$ ls /usr/src | grep nvidia
nvidia-450.51.06

3. 安装一个不知道什么东西，再安一个不知道什么东西（注意需要sudo权限，无权限联系管理员处理）

$ sudo apt-get install dkms
$ sudo dkms install -m nvidia -v 450.51.06 # 注意这个版本号和上一步查到的相对应

4. 完成操作后，docker和Nvidia-SMI均恢复正常