linux D状态进程长时间无动静,主动中断释放,D原因未知
4b(某服务器) 训 33x34 att-10s 内存22G,显存16G,确认当时我就跑了这个gpu进程
出于安全考虑,人名均为化名
# tmux 切到该进程,无动静有半天了
105/105 - 1s - loss: 2.1323e-04 - val_loss: 1.8251e-04
Create new model time: 0.616448163986206
# ctrl+c 中断后资源释放
诊断
内存占用
500G 占 4.6%,不拉满都认为正常
$ ps -eo pid,ppid,%mem,%cpu,cmd --sort=-%mem | head -n 6
PID PPID %MEM %CPU CMD
23420 23236 4.6 28.9 python run_ippo.py
显存占用
驱动问题,在gpu还是cpu跑这个比较玄学,占用那么大显存,原则是上有点问题的
$ gpustat
node04 Wed Apr 24 16:37:52 2024 550.54.14
[0] NVIDIA GeForce RTX 3090 | 86°C, 90 % | 18273 / 24576 MB | zhangsan(15684M) iing(258M) lisi(1110M) lisi(1110M) kde(27M) kde(56M)
[1] NVIDIA GeForce RTX 3090 | 62°C, 88 % | 2891 / 24576 MB | zhangsan(398M) iing(258M) lisi(1110M) lisi(1110M)
[2] NVIDIA GeForce RTX 3090 | 64°C, 90 % | 2795 / 24576 MB | zhangsan(398M) iing(258M) lisi(1062M) lisi(1062M)
[3] NVIDIA GeForce RTX 3090 | 60°C, 48 % | 2795 / 24576 MB | zhangsan(398M) iing(258M) lisi(1062M) lisi(1062M)
进程
查找进程号
$ top | grep D
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23420 zhangsan 20 0 70.386g 0.023t 628168 D 21.1 4.6 1774:45 python
23420 zhangsan 20 0 70.386g 0.023t 628168 D 23.8 4.6 1774:45 python
$ ps aux | grep 23420
zhangsan 23420 28.9 4.6 73769468 24412352 pts/2 Rl+ 4月20 1774:37 python run_ippo.py
网上查,D状态表示进程处于不可中断的睡眠状态,可能是等待磁盘I/O、等待硬件设备等原因。推测是磁盘,但不排除其他原因。比较服务器前面内存、显存都没拉满。确实有内存、显存拉满引发的异常,不过异常会直接中断程序了((ResourceExhaustedError: OOM))
lsof
本想通过 lsof 查看是哪个设备导致的等待,不过发现太多无关文件,不太好定位。最后没有弄明白原因,直接中断任务重新开始了,下面是 lsof 的部分输出。
$ lsof -p 23420
lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/docker/overlay2/224300d3543ff23234fbef0b17256d871e1348d57b9ede8abd3466a2f94b658e/merged
Output information may be incomplete.
lsof: WARNING: can't stat() nsfs file system /run/docker/netns/fbea593b3cbe
Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/aa094d10-9ac1-414d-a1bb-5c9dc61329f8/volumes/kubernetes.io~projected/kube-api-access-
qf8k2
Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/kubelet/pods/230d1279-8261-442b-8e39-88ee70eca235/volumes/kubernetes.io~projected/kube-api-access-
gqj8b
Output information may be incomplete.
# 省略相似内容
lsof: WARNING: can't stat() overlay file system /var/lib/docker/overlay2/5dafb56dd2d21069586611b78b3b30724673ac20a70371663aa85e24736564ea/merged
Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/docker/overlay2/a780ee78727452ab11c8f97aac63d16d64bd8825f9c2683c5f02a1c7c91d95f7/merged
Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/docker/overlay2/39b1b56a3ebc7f12c6b5c9c04668a554696fa07874044f26d18ee952f0ab7887/merged
Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /var/lib/docker/containers/630fc34395f9e010119f76e5084ecd116931165bd33ae724986a4f65e9ba5321/mounts/shm
Output information may be incomplete.
lsof: WARNING: can't stat() nsfs file system /run/docker/netns/711c1e5bb2b4
Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/docker/overlay2/90a63e0149c44191dc1e4d6a7c3b4aa5c57e857c6ec890443ec1b214e225cb8d/merged
Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /var/lib/docker/overlay2/3c33df5756ea8c0d2ff1579ac57e157a034970ce43fef498ed9a0d49ff9acd40/merged
Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /run/snapd/ns
Output information may be incomplete.
lsof: WARNING: can't stat() nsfs file system /run/snapd/ns/gnome-characters.mnt
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 23420 zhangsan cwd DIR 0,54 4096 1121965694 /home/zhangsan/tsc/ippo (192.0.0.2:/home)
python 23420 zhangsan rtd DIR 8,2 4096 2 /
python 23420 zhangsan txt REG 0,54 10668400 1126705228 /home/zhangsan/miniconda3/envs/36/bin/python3.6 (192.0.0.2:/home)
python 23420 zhangsan mem CHR 195,0 710 /dev/nvidia0
python 23420 zhangsan mem CHR 195,255 684 /dev/nvidiactl
python 23420 zhangsan mem CHR 507,0 765 /dev/nvidia-uvm
python 23420 zhangsan mem CHR 195,1 724 /dev/nvidia1
python 23420 zhangsan mem CHR 195,2 736 /dev/nvidia2
python 23420 zhangsan mem CHR 195,3 748 /dev/nvidia3
python 23420 zhangsan DEL REG 0,1 44746876 /dev/zero
python 23420 zhangsan DEL REG 0,1 44717481 /dev/zero
python 23420 zhangsan DEL REG 0,1 44717482 /dev/zero
python 23420 zhangsan DEL REG 0,1 44717484 /dev/zero
python 23420 zhangsan DEL REG 0,1 44710541 /dev/zero
python 23420 zhangsan DEL REG 0,1 44710542 /dev/zero
python 23420 zhangsan DEL REG 0,1 44710544 /dev/zero
python 23420 zhangsan DEL REG 0,1 44680112 /dev/zero
python 23420 zhangsan DEL REG 0,1 44680113 /dev/zero
python 23420 zhangsan DEL REG 0,1 44680115 /dev/zero
python 23420 zhangsan DEL REG 0,1 44699287 /dev/zero
python 23420 zhangsan DEL REG 0,1 44699288 /dev/zero
python 23420 zhangsan DEL REG 0,1 44699290 /dev/zero
python 23420 zhangsan mem REG 8,2 233295448 9973030 /usr/local/cuda-11.1/targets/x86_64-linux/lib/libcusparse.so.11.2.0.275
python 23420 zhangsan mem REG 8,2 190512040 11149081 /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcusolver.so.10.3.0.89
python 23420 zhangsan mem REG 8,2 75343512 9973023 /usr/local/cuda-11.1/targets/x86_64-linux/lib/libcurand.so.10.2.2.74
python 23420 zhangsan mem REG 8,2 235963680 9973019 /usr/local/cuda-11.1/targets/x86_64-linux/lib/libcufft.so.10.3.0.74
python 23420 zhangsan mem REG 8,2 222330208 9973014 /usr/local/cuda-11.1/targets/x86_64-linux/lib/libcublasLt.so.11.2.1.74
python 23420 zhangsan mem REG 8,2 120427896 9973016 /usr/local/cuda-11.1/targets/x86_64-linux/lib/libcublas.so.11.2.1.74
python 23420 zhangsan DEL REG 0,1 44717483 /dev/zero
python 23420 zhangsan DEL REG 0,1 44710543 /dev/zero
python 23420 zhangsan DEL REG 0,1 44680114 /dev/zero
python 23420 zhangsan DEL REG 0,1 44699289 /dev/zero
python 23420 zhangsan mem REG 8,2 162360 9971979 /usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.0
python 23420 zhangsan mem REG 8,2 28392536 9961911 /usr/lib/x86_64-linux-gnu/libcuda.so.550.54.14
python 23420 zhangsan mem REG 0,54 516208 1126707768 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/cityflow.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 5230584 1131025253 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/core/kernels/libtfkernel_sobol_op.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 3993072 1131025420 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/lite/python/interpreter_wrapper/_pywrap_tensorflow_interpreter_wrapper.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 114576 1131023533 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_toco_api.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 948952 1131025400 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/lite/experimental/microfrontend/python/ops/_audio_microfrontend_op.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 269424 1131023517 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_python_op_gen.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1440544 1131023539 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_mlir.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1299144 1131023549 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_debug_events_writer.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 102224 1131023534 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_util_port.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 101848 1131023518 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_stacktrace_handler.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1268768 1131023522 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_quantize_training.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1732752 1131023514 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_tfprof.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1374032 1131023519 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_tf_cluster.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1291752 1131023521 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_tf_optimizer.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 117504 1130978688 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5l.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 129752 1130978702 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5o.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 47224 1130978686 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5pl.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 212792 1130978689 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5fd.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 67968 1130978705 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5i.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 187072 1130978680 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5g.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 142280 1130978699 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5f.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 92856 1130978693 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5ds.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 146224 1130978685 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5d.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 55424 1130978701 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/_proxy.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 63816 1130978681 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5ac.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 388584 1130978695 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5p.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 125680 1130978707 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5s.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 158544 1130978691 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5a.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 58200 1130978684 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5z.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 60024 1130978700 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/utils.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 549128 1130978697 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5t.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 76376 1130978696 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5r.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 97096 1130978682 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/_conv.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 150912 1130978703 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/_objects.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 257640 1130978683 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/defs.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 125880 1130978698 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/h5.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 87848 1130978713 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/.libs/libz-a147dcb0.so.1.2.3 (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 37712 1130978710 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/.libs/libaec-2147abcd.so.0.0.4 (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 16792 1130978711 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/.libs/libsz-1c7dd0cf.so.2.0.1 (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 186152 1130978712 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/.libs/libhdf5_hl-db841637.so.100.1.1 (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 4187280 1130978709 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/.libs/libhdf5-beac1db3.so.103.0.0 (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 64432 1130978694 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/h5py/_errors.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 108000 1126705858 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/testing.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 105528 1126705854 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/json.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 585160 1126705954 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/parsers.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 342312 1126705924 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/reduction.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1235240 1126705979 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/groupby.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 190704 1126705867 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/window/indexers.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 479360 1126705940 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/window/aggregations.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 282776 1126705907 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/internals.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 239328 1126705891 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/writers.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 303816 1126705913 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/reshape.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 940432 1126705971 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/sparse.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 2822696 1126705990 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/join.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 618728 1126705957 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/index.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 245144 1126705895 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/ops.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 200056 1126705874 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/hashing.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 195632 1126705870 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslib.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 629976 1126705962 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/lib.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1662408 1126705987 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/algos.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 71416 1126705842 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/ops_dispatch.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 254128 1126705899 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslibs/vectorized.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 473064 1126705935 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslibs/period.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 82296 1126705846 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/properties.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 446000 1126705927 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslibs/strptime.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 298608 1126705910 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslibs/fields.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 515648 1126705949 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslibs/timestamps.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 499016 1126705945 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslibs/timedeltas.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1121952 1126705976 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslibs/offsets.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 471072 1126705932 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslibs/parsing.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 94368 1126705851 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslibs/ccalendar.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 327512 1126705920 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslibs/tzconversion.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 258960 1126705902 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslibs/timezones.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 64240 1126705839 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslibs/np_datetime.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 233344 1126705881 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslibs/nattype.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 314160 1126705916 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslibs/conversion.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 149544 1126705862 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslibs/dtypes.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 227112 1126705878 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/missing.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 664024 1126705967 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/hashtable.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1387144 1126705984 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/interval.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 953992 1122630212 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 678000 1124448346 /home/zhangsan/miniconda3/envs/36/lib/libssl.so.1.1 (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1273328 1131023515 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_device_lib.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 247808 1122630200 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1330080 1131024939 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/data/experimental/service/_pywrap_server_lib.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 101648 1131023545 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_py_func.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 102968 1131023542 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensor_float_32_execution.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1324296 1131023526 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_parallel_device.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1863296 1131023537 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_checkpoint_reader.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1370672 1131023547 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_record_io.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 5410256 1131023974 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/profiler/internal/_pywrap_profiler.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1388720 1131023553 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_file_io.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 8,2 27112 22807808 /lib/x86_64-linux-gnu/libuuid.so.1.3.0
python 23420 zhangsan mem REG 0,54 76856 1122630171 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/mmap.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 46408 1122630157 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 150616 1122630189 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1309088 1131023530 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_events_writer.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 57520 1126705836 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/indexing.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 55448 1126705833 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/pandas/_libs/tslibs/base.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 115960 1122630182 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_multibytecodec.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 293696 1122630203 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1267872 1131023532 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_op_def_registry.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 114944 1131025063 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/framework/fast_tensor_util.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1297720 1131023977 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/profiler/internal/_pywrap_traceme.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 324376 1131023551 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_dtypes.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 101608 1131023516 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_bfloat16.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 442600 1131023538 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_py_exception_registry.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 136912 1131023536 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_utils.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 170912 1130977549 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/wrapt/_wrappers.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 942888 1131023552 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_tf_stack.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 1982240 1131023529 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_tf_session.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 2169280 1131023531 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_tfe.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 2548904 1130977155 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 14408 1130977142 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/google/protobuf/internal/_api_implementation.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 49648 1122630162 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 120848 1122630184 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_csv.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 246568 1122630199 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 47816 1122630160 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 8,2 540184 9973007 /usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudart.so.11.1.74
python 23420 zhangsan mem REG 0,54 17981480 1121835729 /home/zhangsan/miniconda3/envs/36/lib/libstdc++.so.6.0.29 (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 39413472 1130978951 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/libtensorflow_framework.so.2 (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 699303912 1131023525 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 27080 1122630150 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 14093170 1126707657 /home/zhangsan/miniconda3/envs/36/lib/libmkl_vml_avx512.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 61346160 1126707807 /home/zhangsan/miniconda3/envs/36/lib/libmkl_avx512.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 13049502 1126707649 /home/zhangsan/miniconda3/envs/36/lib/libmkl_intel_lp64.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 59196279 1126707774 /home/zhangsan/miniconda3/envs/36/lib/libmkl_intel_thread.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 74028346 1126707880 /home/zhangsan/miniconda3/envs/36/lib/libmkl_core.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 852152 1126707122 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/numpy/random/_generator.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 71392 1126707085 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/numpy/random/_sfc64.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 87704 1126707089 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/numpy/random/_pcg64.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 105928 1126707095 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/numpy/random/_philox.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 119672 1126707098 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/numpy/random/_mt19937.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 386080 1126707115 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/numpy/random/_bounded_integers.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 56928 1122630166 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 33904 1122630152 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 386992 1122630207 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 184568 1122630194 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 3270744 1126703647 /home/zhangsan/miniconda3/envs/36/lib/libcrypto.so.1.1 (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 82352 1122630173 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 80224 1122630172 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 268848 1126707112 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/numpy/random/_common.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 219432 1126707109 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/numpy/random/bit_generator.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 732064 1126707119 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/numpy/random/mtrand.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 92688 1126707092 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/numpy/fft/_pocketfft_internal.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 818728 1122630211 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 213984 1122560279 /home/zhangsan/miniconda3/envs/36/lib/liblzma.so.5.4.5 (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 101544 1122630180 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 135152 1122630187 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 124744 1121835323 /home/zhangsan/miniconda3/envs/36/lib/libz.so.1.2.13 (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 108232 1122630181 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 195856 1126707106 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 26568 1126707076 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 74256 1122630170 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 58344 1122630167 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 164680 1126707102 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/numpy/core/_multiarray_tests.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 3723760 1126707129 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/numpy/core/_multiarray_umath.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 146456 1126705782 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/mkl/_py_mkl_service.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 475272 1121785837 /home/zhangsan/miniconda3/envs/36/lib/libgcc_s.so.1 (192.0.0.2:/home)
python 23420 zhangsan mem REG 8,2 31680 22811535 /lib/x86_64-linux-gnu/librt-2.27.so
python 23420 zhangsan mem REG 0,54 428160 1122630209 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 2624896 1126705661 /home/zhangsan/miniconda3/envs/36/lib/libiomp5.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 7100733 1126707601 /home/zhangsan/miniconda3/envs/36/lib/libmkl_rt.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 8,2 4795856 9970368 /usr/lib/locale/locale-archive
python 23420 zhangsan mem REG 8,2 2030928 22806611 /lib/x86_64-linux-gnu/libc-2.27.so
python 23420 zhangsan mem REG 8,2 144976 22811475 /lib/x86_64-linux-gnu/libpthread-2.27.so
python 23420 zhangsan mem REG 8,2 1700792 22806615 /lib/x86_64-linux-gnu/libm-2.27.so
python 23420 zhangsan mem REG 8,2 10592 22811537 /lib/x86_64-linux-gnu/libutil-2.27.so
python 23420 zhangsan mem REG 8,2 14560 22806614 /lib/x86_64-linux-gnu/libdl-2.27.so
python 23420 zhangsan mem REG 8,2 179152 22806544 /lib/x86_64-linux-gnu/ld-2.27.so
python 23420 zhangsan mem REG 0,54 41520 1122630155 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 17008 1126705779 /home/zhangsan/miniconda3/envs/36/lib/python3.6/site-packages/mkl/_mklinit.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 151576 1122630190 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 414664 1122630208 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 311720 1122630206 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 122440 1122630185 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 50664 1124448844 /home/zhangsan/miniconda3/envs/36/lib/libffi.so.7.1.0 (192.0.0.2:/home)
python 23420 zhangsan mem REG 0,54 47376 1122630159 /home/zhangsan/miniconda3/envs/36/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so (192.0.0.2:/home)
python 23420 zhangsan mem REG 8,2 26376 10229122 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
python 23420 zhangsan 0u CHR 136,2 0t0 5 /dev/pts/2
python 23420 zhangsan 1u CHR 136,2 0t0 5 /dev/pts/2
python 23420 zhangsan 2u CHR 136,2 0t0 5 /dev/pts/2
python 23420 zhangsan 3r CHR 1,3 0t0 6 /dev/null
python 23420 zhangsan 4w FIFO 0,13 0t0 44719496 pipe
python 23420 zhangsan 5u a_inode 0,14 0 11836 [eventfd]
python 23420 zhangsan 6r FIFO 0,13 0t0 44675950 pipe
python 23420 zhangsan 7w FIFO 0,13 0t0 44675950 pipe
python 23420 zhangsan 8r FIFO 0,13 0t0 44675951 pipe
python 23420 zhangsan 9w FIFO 0,13 0t0 44675951 pipe
python 23420 zhangsan 10u CHR 195,255 0t0 684 /dev/nvidiactl
python 23420 zhangsan 11u CHR 507,0 0t0 765 /dev/nvidia-uvm
python 23420 zhangsan 12u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 13u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 14u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 15u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 16u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 17u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 18u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 19u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 20u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 21u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 22u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 23u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 24u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 25u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 26u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 27u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 28u unix 0x0000000000000000 0t0 44674030 @cuda-uvmfd-4026531836-23420@ type=SEQPACKET
python 23420 zhangsan 29u a_inode 0,14 0 11836 [eventfd]
python 23420 zhangsan 30u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 31u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 32u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 33u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 34u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 35u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 36u a_inode 0,14 0 11836 [eventfd]
python 23420 zhangsan 37u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 38u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 39u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 40u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 41u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 42u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 43u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 44u CHR 195,0 0t0 710 /dev/nvidia0
python 23420 zhangsan 45r FIFO 0,13 0t0 44699293 pipe
python 23420 zhangsan 46w FIFO 0,13 0t0 44699293 pipe
python 23420 zhangsan 47u a_inode 0,14 0 11836 [eventfd]
python 23420 zhangsan 48u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 49u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 50u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 51u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 52u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 53u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 54u a_inode 0,14 0 11836 [eventfd]
python 23420 zhangsan 55u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 56u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 57u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 58u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 59u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 60u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 61u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 62u CHR 195,1 0t0 724 /dev/nvidia1
python 23420 zhangsan 63u a_inode 0,14 0 11836 [eventfd]
python 23420 zhangsan 64u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 65u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 66u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 67u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 68u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 69u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 70u a_inode 0,14 0 11836 [eventfd]
python 23420 zhangsan 71u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 72u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 73u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 74u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 75u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 76u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 77u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 78u CHR 195,2 0t0 736 /dev/nvidia2
python 23420 zhangsan 79u a_inode 0,14 0 11836 [eventfd]
python 23420 zhangsan 80u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 81u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 82u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 83u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 84u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 85u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 86u a_inode 0,14 0 11836 [eventfd]
python 23420 zhangsan 87u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 88u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 89u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 90u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 91u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 92u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 93u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 94u CHR 195,3 0t0 748 /dev/nvidia3
python 23420 zhangsan 95r REG 0,5 0 44674025 /proc/23420/cmdline