HPC install software on python virtual environment without root right and use X graph (windows) to display

1. Using Python Virtual Environment to install new softwares without root right.

 

Please try to use Python virtualenv to setup and manage Python environments

 

https://media.readthedocs.org/pdf/virtualenv/latest/virtualenv.pdf

 

For example to use Python 3.6.3 with PyTorch 0.2.0_3, I’m working in the folder /home/wang/pyenv

 

(1) To create a new folder

 

 mkdir py3.6.3

 

(2)

module load pytorch/python3.6/0.2.0_3

(3) 

virtualenv --system-site-packages py3.6.3

 

(4)

source py3.6.3/bin/activate

(5)

 pip install opencv-python Pillow pywavelets scikit-learn

 

Now every time after run

 

module load pytorch/python3.6/0.2.0_3
source /home/wang/pyenv/py3.6.3/bin/activate

 

you’ll use this environment. 

 

[wang@c17-04 ~]$ module load pytorch/python3.6/0.2.0_3

[wang@c17-04 ~]$ source /home/wang/pyenv/py3.6.3/bin/activate

(py3.6.3) [wang@c17-04 ~]$ python -c "import torch; print(torch.__version__); print(torch.__file__)"

0.2.0_3

/share/apps/pytorch/0.2.0_3/python3.6/lib/python3.6/site-packages/torch/__init__.py

(py3.6.3) [wang@c17-04 ~]$ python -c "import cv2; print(cv2.__version__); print(cv2.__file__)"

3.3.0

/home/wang/pyenv/py3.6.3/lib/python3.6/site-packages/cv2/cv2.cpython-36m-x86_64-linux-gnu.so

(py3.6.3) [wang@c17-04 ~]$

 

 

 

2. Use X graph (windows) to display

 

Inside NYU campus, please directly login to prince.hpc.nyu.edu without through HPC gateways with command

 

ssh -Y prince.hpc.nyu.edu
 

When you are working off campus, please try to use HPC gateways is gw.hpc.nyu.edu, /home on new gateway is mounted as read only, users can not write any data there.  HPC gateways are mainly for setup ssh tunneling, after sshtunneling session is running, users can directly login to prince cluster from their local computers. 

 

For Mac, please follow the instructions on HPC wiki to setup ssh tunneling

 

https://wikis.nyu.edu/pages/viewpage.action?pageId=85395532

 

(1) Open an iTerm, run command

 

ssh hpcgwtunnel

 

to setup ssh tunneling through gateway.

 

(2) Keep iTerm window in (1) open, open a new iTerm run command

 

ssh prince -Y

you should be able to directly login to prince from your local computer as long as ssh tunneling in step (1) is running, X11 forwarding is enabled.

 

scp is similar as ssh.

 

For Windows with Putty for ssh login

 

https://wikis.nyu.edu/display/NYUHPC/Accessing+HPC+clusters+from+Windows+workstation

 

For Windows with SCP for data transfer

 

https://wikis.nyu.edu/pages/viewpage.action?pageId=84612833

 

Wireshark 2.6.3. is available with Singularity image file 

 

/beegfs/work/public/singularity/wireshark-2.6.3.simg

 

Please launch wireshark with command

 

singularity exec /beegfs/work/public/singularity/wireshark-2.6.3.simg wireshark

 

[wang@c17-04 ~]$ singularity exec /beegfs/work/public/singularity/wireshark-2.6.3.simg which wireshark

/usr/bin/wireshark

[wang@c17-04 ~]$ singularity exec /beegfs/work/public/singularity/wireshark-2.6.3.simg wireshark --version

Wireshark 2.6.3 (Git v2.6.3 packaged as 2.6.3-1~18.04.0)

 

Copyright 1998-2018 Gerald Combs <gerald@wireshark.org> and contributors.

License GPLv2+: GNU GPL version 2 or later <http://www.gnu.org/licenses/old-licenses/gpl-2.0.html>

This is free software; see the source for copying conditions. There is NO

warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

 

Compiled (64-bit) with Qt 5.9.5, with libpcap, with POSIX capabilities (Linux),

with libnl 3, with GLib 2.56.2, with zlib 1.2.11, with SMI 0.4.8, with c-ares

1.14.0, with Lua 5.2.4, with GnuTLS 3.5.18, with Gcrypt 1.8.1, with MIT

Kerberos, with MaxMind DB resolver, with nghttp2 1.30.0, with LZ4, with Snappy,

with libxml2 2.9.4, with QtMultimedia, with SBC, with SpanDSP, without bcg729.

 

Running on Linux 3.10.0-514.10.2.el7.x86_64, with Intel(R) Xeon(R) CPU E5-2690

v4 @ 2.60GHz (with SSE4.2), with 128826 MB of physical memory, with locale C,

with libpcap version 1.8.1, with GnuTLS 3.5.18, with Gcrypt 1.8.1, with zlib

1.2.11, binary plugins supported (0 loaded).

 

Built using gcc 7.3.0.

[wang@c17-04 ~]$

singularity exec /beegfs/work/public/singularity/wireshark-2.6.3.simg wireshark

 

 

 

 Solution:

 

https://uisapp2.iu.edu/confluence-prd/pages/viewpage.action?pageId=280461906

Mac OS X

1. Install XQuartz on your Mac, which is the official X server software for Mac
2. Run Applications > Utilities > XQuartz.app

3. Right click on the XQuartz icon in the dock and select Applications > Terminal.  This should bring up a new xterm terminal windows.

4. In this xterm windows, ssh into the linux system of your choice using the -X argument (secure X11 forwarding).  For example, to log into hulk.soic.indiana.edu you would run something like:
   
        ssh -Y username@hulk.soic.indiana.edu
   
   
5. Once you are logged into the linux system, you can just run the GUI program of your choice (ie. matlab, mathematics, etc) and it will display on your Mac.

 

https://wikis.nyu.edu/display/ADRC/Enable+X11+Forwarding+on+Mac+OS+X

Enable X11 Forwarding on Mac OS X

If you own a recent Mac computer and you realize that you can't start any GUI app from the cluster, even when you used the -X option with ssh; then you may have to enable X11 Forwarding on MacOS.

Step-by-step guide

1. By default, X11 forwarding is not enabled on Mac Leopard, and from Mountain Lion on, Apple decided not to ship X11 with the OS. If that latter is your case, then first you will have to install XQuartz from here:  http://xquartz.macosforge.org.

2. Once you have it installed, you should be able to connect to prince with X11.

   ssh -X <NetID>@prince.hpc.nyu.edu