webdriver.close() quit() 批量kill进程 内存耗尽的解决办法

 

问题现象：

 

　　shell窗口卡，换IP的登录窗，不开；

猜测：

　　内存耗尽

 

spider_url,py

driver = webdriver.PhantomJS(
    executable_path='/usr/local/phantomjs/bin/phantomjs')
driver.get(url)
time.sleep(1)
page_source = driver.page_source
driver.close()


源码

def close(self):
    """
    Closes the current window.

    :Usage:
        driver.close()
    """
    self.execute(Command.CLOSE)

def quit(self):
    """
    Quits the driver and closes every associated window.

    :Usage:
        driver.quit()
    """
    try:
        self.execute(Command.QUIT)
    finally:
        self.stop_client()

 

修改爬虫代码:

　　

driver.quit()




查看系统信息：

[root@hadoop1 ~]# free -g
total used free shared buff/cache available
Mem: 15 14 0 0 0 0
Swap: 7 7 0

 

top - 18:40:01 up 13 days, 3:10, 2 users, load average: 97.35, 90.95, 101.05
Tasks: 899 total, 5 running, 893 sleeping, 0 stopped, 1 zombie
%Cpu(s): 7.0 us, 38.9 sy, 0.0 ni, 4.5 id, 49.5 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 16203484 total, 136668 free, 15537644 used, 529172 buff/cache
KiB Swap: 8191996 total, 0 free, 8191996 used. 73852 avail Mem

KiB Mem : 16203484 total, 136796 free, 15537716 used, 528972 buff/cache
17409 root 20 0 3024380 22576 0 D 36.0 0.1 33:14.32 phantomjs
7721 root 20 0 3396256 79112 0 S 13.0 0.5 0:53.85 python
8204 root 20 0 1261576 6192 364 D 9.0 0.0 0:04.38 phantomjs
32149 root 20 0 2867240 29636 0 D 8.0 0.2 12:16.97 phantomjs
23292 root 20 0 2858092 28044 68 D 7.9 0.2 24:55.04 phantomjs
7990 root 20 0 2762952 11924 0 D 7.5 0.1 0:08.42 phantomjs
32081 root 20 0 3366480 50012 0 D 7.3 0.3 31:55.27 phantomjs
11454 root 20 0 3302384 46948 0 D 6.2 0.3 27:31.83 phantomjs
8203 root 20 0 1261540 5996 260 D 5.4 0.0 0:02.52 phantomjs
15485 root 20 0 3627320 43680 428 D 5.1 0.3 28:59.12 phantomjs
8227 root 20 0 198472 688 284 D 5.0 0.0 0:02.28 phantomjs
8205 root 20 0 1261576 6016 236 D 4.6 0.0 0:02.05 phantomjs
31746 root 20 0 3287492 32200 0 D 4.5 0.2 10:20.00 phantomjs
7639 root 20 0 3153564 36620 0 D 4.1 0.2 0:22.70 phantomjs
30213 root 20 0 3249684 52196 0 D 3.8 0.3 34:24.91 phantomjs
31987 root 20 0 3102684 44156 0 D 3.8 0.3 15:50.06 phantomjs
9247 root 20 0 3088060 34940 0 D 3.6 0.2 45:02.70 phantomjs
26481 root 20 0 3031160 35268 112 D 3.5 0.2 58:33.10 phantomjs
19825 root 20 0 3127540 42592 0 D 3.4 0.3 34:37.43 phantomjs
19196 root 20 0 3121804 41776 188 D 3.2 0.3 25:07.63 phantomjs
21684 root 20 0 3157068 24540 0 D 3.2 0.2 35:58.68 phantomjs
20445 root 20 0 3101172 44876 212 D 2.9 0.3 39:05.05 phantomjs
24628 root 20 0 3049636 26620 0 D 2.9 0.2 32:25.60 phantomjs
7279 root 20 0 3026656 23160 60 D 2.7 0.1 36:16.57 phantomjs
31636 root 20 0 3431992 50664 0 S 2.7 0.3 28:03.55 phantomjs
19543 root 20 0 3013740 33872 0 D 2.6 0.2 23:07.82 phantomjs
7101 root 20 0 3123908 28004 0 D 2.6 0.2 39:26.25 phantomjs
19542 root 20 0 3093216 35980 0 D 2.6 0.2 28:10.17 phantomjs
19965 root 20 0 3163648 38592 0 D 2.5 0.2 27:56.73 phantomjs
32235 root 20 0 3156676 35300 0 D 2.5 0.2 10:18.39 phantomjs
23132 root 20 0 3088976 33640 0 D 2.3 0.2 25:47.01 phantomjs
92 root 20 0 0 0 0 D 2.1 0.0 192:21.37 kswapd0

 

 

定时任务执行多线程脚本后，phantomjs进程没有释放资源，退出

 

 

解决办法：

　　批量杀死该进程，

 

 

实验

vim tmpxx.py

import time
for i in range(1,100,1):
time.sleep(1)
print(i)

 

ps -aux | grep tmpxx.py | grep -v grep
ps -aux | grep tmpxx.py | grep -v grep |cut -c 9-15
ps -aux | grep tmpxx.py | grep -v grep |cut -c 9-15 | xargs kill -9

检验

实验成功

 

 

ps -aux | grep phantomjs | grep -v grep |cut -c 9-15 | xargs kill -9

 

检验

 

 

top - 18:54:19 up 13 days, 3:25, 3 users, load average: 10.27, 48.42, 78.57
Tasks: 197 total, 1 running, 196 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16203484 total, 15612264 free, 199696 used, 391524 buff/cache
KiB Swap: 8191996 total, 8170176 free, 21820 used. 15558816 avail Mem

 

 

 load average: 10.27, 48.42, 78.57 说明实验成功

 

[root@hadoop1 ~]# free -g
total used free shared buff/cache available
Mem: 15 0 14 0 0 14
Swap: 7 0 7

 

ok

 

 

Last login: Mon Dec 11 13:41:52 2017 from 192.168.2.96
[root@hadoop1 ~]# ssh root@192.168.2.212 'll -as'
Connection closing...Socket close.

Connection closed by foreign host.

Disconnected from remote host(192.168.2.51) at 13:50:15.

 

 

 

ssh  在往另外一台机器发送命令

 

 

Last login: Mon Dec 11 13:52:36 2017 from 192.168.2.96
[root@hadoop1 ~]# ssh root@192.168.2.212 'ps -aux | grep phantomjs | grep -v grep |cut -c 9-15 | xargs kill -9'