【异常修复】Docker启动容器时无法使用GPU

硬件配置

显卡：Tesla V100

问题呈现

  1. 某次重启后，发现服务器nvidia-docker无法start已有的部分容器
$ docker start ******
Error response from daemon: failed to create shim: OCI runtime create failed
: container_linux.go:380: starting container process caused
: process_linux.go:545: container init caused
: Running hook #0:: error running hook: exit status 1, stdout: , stderr
: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown
Error: failed to start containers: ******
  1. 为排查问题的可能原因，尝试另行启动容器，发现启动时无法使用GPU
$ docker run -it --gpus all [Image_Name]:[Image_Tag]
docker: 
Error response from daemon: failed to create shim: OCI runtime create failed
: container_linux.go:380: starting container process caused
: process_linux.go:545: container init caused
: Running hook #0:: error running hook: exit status 1, stdout: , stderr
: nvidia-container-cli: initialization error: nvml error: driver not loaded: unknown.
  1. 为了进一步排查问题的可能原因，使用nvidia-smi命令查看显卡情况，发现异常
$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. 
Make sure that the latest NVIDIA driver is installed and running.

问题定位

docker报错中包含nvml error: driver not loaded: unknown，因此问题产生原因可能是显卡驱动被挤掉了

解决方案

  1. 查验现状
    用nvcc -V确认显卡驱动存在
$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0
  1. 查看已安装驱动的版本信息
$ ls /usr/src | grep nvidia
nvidia-450.51.06
  1. 安装一个不知道什么东西，再安一个不知道什么东西（注意需要sudo权限，无权限联系管理员处理）
$ sudo apt-get install dkms
$ sudo dkms install -m nvidia -v 450.51.06 # 注意这个版本号和上一步查到的相对应
  1. 完成操作后，docker和Nvidia-SMI均恢复正常
posted @ 2022-04-21 21:20  TheError  阅读(4857)  评论(0)  编辑  收藏  举报