docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

docker使用--gpus all报错：

官方安装指导：https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

在网上查询了很多文章，总结起来就是要安装nvidia-container-toolkit或nvidia-container-runtime(包含nvidia-container-toolkit)
但是尴尬的是怎么都安装不了nvidia-container-toolkit，一直显示 ** E: Unable to locate package nvidia-container-toolkit**

网上的解决方案：

• distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
• curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
  这个大家应该比较熟，老版本的docker安装都会使用这个进行添加GPG key
• curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | sudo tee /etc/yum.repos.d/nvidia-docker.repo
  或者curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
• sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
• sudo systemctl restart docker

上面的方法我都进行了尝试，这里要注意第三步，centos和Ubuntu命令不一样！

補充：華爲雲的centos環境建議直接安裝nvidia-container-runtime，我在使用的過程中安裝nvidia-container-toolkit不知道爲什麼在重啓後沒有生效

使用上面的命令我还是安装不了，最后解决的过程记录如下：

• 更改系统源为阿里的镜像源
• distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
• curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - 正常会显示OK
• curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list 我的系统是Ubuntu18.04
• sudo apt-get update 这一步要保证没得问题，我的显示有几个源重复配置，然后我就将其(sudo vim nvidia-docker.list)注释掉
• sudo apt-get install nvidia-container-toolkit
• sudo systemctl restart docker

总结

实现路径是一样的，就是更新源那么简单吗？实际上公司的网络非常差很不稳定，导致很多步骤不能正常执行，如sudo apt-get update一会可以正常执行，一会报错。

安装后报错：

/sbin/ldconfig.real: /usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn.so.8 is not a symbolic link
/sbin/ldconfig.real: /usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_train.so.8 is not a symbolic link
/sbin/ldconfig.real: /usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8 is not a symbolic link
/sbin/ldconfig.real: /usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8 is not a symbolic link
/sbin/ldconfig.real: /usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8 is not a symbolic link
/sbin/ldconfig.real: /usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_train.so.8 is not a symbolic link
/sbin/ldconfig.real: /usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8 is not a symbolic link
 

解决方案

$ sudo ln -sf /usr/local/cuda/targets/x86_64-linux/lib/libcudnn.so.8.4.0 /usr/local/cuda/targets/x86_64-linux/lib/libcudnn.so.8
$ sudo ln -sf /usr/local/cuda/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.4.0 /usr/local/cuda/targets/x86_64-linux/lib/libcudnn_adv_train.so.8
$ sudo ln -sf /usr/local/cuda/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.4.0 /usr/local/cuda/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8
$ sudo ln -sf /usr/local/cuda/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.4.0 /usr/local/cuda/targets/x86_64-linux/lib/libcudnn_ops_train.so.8
$ sudo ln -sf /usr/local/cuda/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.4.0 /usr/local/cuda/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8
$ sudo ln -sf /usr/local/cuda/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.4.0 /usr/local/cuda/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8
$ sudo ln -sf /usr/local/cuda/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.4.0 /usr/local/cuda/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8

完成后记得重启docker