libcublas.so.10.0: cannot open shared object file in singularity, however, it indeed exists in the directory.

When I execute the code out of the singularity (/rigel/opt/tshark/tshark_python3.sif), everything is correct.

 

 

 

However, when I execute it in singularity, it always shows me that libcublas.so.10.0: cannot open shared object file in singularity, even though it exists in the directory (/cm/shared/apps/cuda10.0/toolkit/10.0.130/targets/x86_64-linux/lib/libcublas.so.10.0).

$singularity exec --nv /rigel/opt/tshark/tshark_python3.sif python

>import tensorflow as tf

>tf.test.is_gpu_available()

 

 

 

 

 

 

Even I add "the directory path", the problem still exists.

$env SINGULARITYENV_APPEND_PATH=/cm/shared/apps/cuda10.0/toolkit/10.0.130/targets/x86_64-linux/lib/ SINGULARITY_ENV_LD_LIBRARY_PATH=$LD_LIBRARY_PATH singularity exec --nv /rigel/opt/tshark/tshark_python3.sif python

 

After several days attempting, I found the reason was that: in the singularity, the directory (/cm/shared/apps/) contains 'libcublas.so.10.0' cann't be seen in the singularity.

 

 

Knowing the reason, it's easy to solve.

Solution: just using --bind.

> singularity exec --nv --bind /cm/shared/apps /rigel/opt/tshark/tshark_python3.sif python

>import tensorflow as tf

>tf.test.is_gpu_available()

 

 

 

 

 

Error: pip install 

ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device

TMPDIR=./pip_cache pip install --upgrade tensorflow_gpu==2.0.0 --user

http://naoko.github.io/pip-install-no-space-left/