selenium部署

一.部署单节点服务

1.docker部署

sudo docker run -d -p 9000:4444 --shm-size=2g selenium/standalone-chrome

2.查看是否部署成功

• 云服务器使用时，开放4444端口还是不能远程连接，改用9000就可以，暂不清楚原因

3. 测试代码如下

• 需提前安装selenium： pip install selenium

from selenium import webdriver


def func(ip, port):
    options = webdriver.ChromeOptions()
    options.add_argument("headless")
    options.add_argument('--no-sandbox')
    options.add_argument('--disable-gpu')
    options.add_argument('--disable-dev-shm-usage')
    ce = f"http://{ip}:{port}/wd/hub"

    with webdriver.Remote(command_executor=ce, options=options) as browser:
        browser.get("http://baidu.com")
        print(browser.title)


if __name__ == '__main__':
    func("xxx", 9000)  # 改为自己服务器的端口

执行结果：

 

 

 

二、部署多节点服务

1.创建docker局域网

docker network create grid

• 方便dockers容器之间互相访问

2. 创建hub

docker run -tid --name selenium-hub -h selenium-hub -e GRID_MAX_SESSION=10 --memory 1g --net grid --memory-swap -1 -p 9010:4444 selenium/hub 

3. 创建node

node1:

docker run -tid --name selenium-node-chrome -h selenium-node-chrome -e SE_EVENT_BUS_HOST=selenium-hub -e SE_EVENT_BUS_PUBLISH_PORT=4442 -e SE_EVENT_BUS_SUBSCRIBE_PORT=4443 --memory 1g --memory-swap -1 --net grid selenium/node-chrome

node2:

docker run -tid --name selenium-node2-chrome -h selenium-node2-chrome -e SE_EVENT_BUS_HOST=selenium-hub -e SE_EVENT_BUS_PUBLISH_PORT=4442 -e SE_EVENT_BUS_SUBSCRIBE_PORT=4443 --memory 1g --memory-swap -1 --net grid selenium/node-chrome

node3:

docker run -tid --name selenium-node3-chrome -h selenium-node3-chrome -e SE_EVENT_BUS_HOST=selenium-hub -e SE_EVENT_BUS_PUBLISH_PORT=4442 -e SE_EVENT_BUS_SUBSCRIBE_PORT=4443 --memory 1g --memory-swap -1 --net grid selenium/node-chrome

 

创建成功检查：

 

 端口号为selenium-hub服务的端口号：

 

 4.执行测试：

import time
from threading import Thread

from selenium import webdriver


def func(idx, ip, port):
    options = webdriver.ChromeOptions()
    options.add_argument("headless")
    options.add_argument('--no-sandbox')
    options.add_argument('--disable-gpu')
    options.add_argument('--disable-dev-shm-usage')
    options.add_argument('--allowed-ips')
    ce = f"http://{ip}:{port}/wd/hub"

    with webdriver.Remote(command_executor=ce, options=options) as browser:
        browser.get("http://baidu.com")

        print(browser.title + ', idx: ' + str(idx) + " " + str(time.time()))
        time.sleep(5)
        print('end' + ', idx: ' + str(idx) + " " + str(time.time()))


def run(ip, port):
    for i in range(5):
        t = Thread(target=func, args=(i, ip, port))
        t.start()
    print("all end")
    

if __name__ == '__main__':
    run('xxx', 9010)

执行结果：

 

•  三个节点并行处理了任务，大功告成！

 

qs:

1.selenium 服务访问时，访问几次就访问不了。查看selenium-hub的日志的时候报错：Exited (137) 

原因：内存分配太少

通过journalctl查看oom日志如下：

journalctl -k | grep -i -e memory -e oom

 

解决方法：需修改创建容器中的--memory参数

docker run -tid --name selenium-hub -h selenium-hub -e GRID_MAX_SESSION=10 --memory 1g --net grid --memory-swap -1 -p 9010:4444 selenium/hub 

 

三、docker-compose部署

docker-compose.yml

# To execute this docker-compose yml file use `docker-compose -f docker-compose-v2.yml up`
# Add the `-d` flag at the end for detached execution
# To stop the execution, hit Ctrl+C, and then `docker-compose -f docker-compose-v2.yml down`
version: '2'
services:
  chrome:
    image: selenium/node-chrome
    shm_size: 2gb
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    ports:
      - "6900:5900"

  edge:
    image: selenium/node-edge
    shm_size: 2gb
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    ports:
      - "6901:5900"

  firefox:
    image: selenium/node-firefox
    shm_size: 2gb
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    ports:
      - "6902:5900"

  selenium-hub:
    image: selenium/hub
    ports:
      - "4442:4442"
      - "4443:4443"
      - "9020:4444"