docker容器内运行pytorch多gpu报错 RuntimeError: NCCL Error 2: unhandled system error

打开nccl的错误信息：

export NCCL_DEBUG=WARN

 

错误1.docker容器内运行pytorch多gpu报错 RuntimeError: NCCL Error 2: unhandled system error
在启动容器的时候加上 -e NVIDIA_VISIBLE_DEVICES=0,1,2,3

 

docker run --runtime=nvidia --net="host" -e NVIDIA_VISIBLE_DEVICES=0,1,2,3 --shm-size 8g -it huangzc/reid:v1 /bin/bash

 

错误2.RuntimeError: DataLoader worker (pid 53617) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.

 

 

 

启动容器的时候增加交换内存 --shm-size 8g