关于gunicorn与异步兼容性问题：AttributeError: module 'select' has no attribute 'epoll'

关于gunicorn与异步兼容性问题：AttributeError: module 'select' has no attribute 'epoll'

背景：

  • 介绍：
  在使用gunicorn、Flask & flask-sockets 部署，实现websocket协议中同类消息阻塞，不同类消息不阻塞场景。
  • 异常：
    [2024-01-15 10:22:16 +0800] [31655] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/home/venv/lib/python3.10/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/home/venv/lib/python3.10/site-packages/gunicorn/workers/ggevent.py", line 162, in init_process
    super().init_process()
  File "/home/venv/lib/python3.10/site-packages/gunicorn/workers/base.py", line 133, in init_process
    self.load_wsgi()
  File "/home/venv/lib/python3.10/site-packages/gunicorn/workers/base.py", line 142, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/home/venv/lib/python3.10/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/home/venv/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
    return self.load_wsgiapp()
  File "/home/venv/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/home/venv/lib/python3.10/site-packages/gunicorn/util.py", line 331, in import_app
    mod = importlib.import_module(module)
  File "/home/nlu_common/miniconda3/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
 。。。。。。包含很多具体代码，没有意义。。。。。。
  File "/home/venv/lib/python3.10/site-packages/dns/resolver.py", line 38, in <module>
    import dns.query
  File "/home/venv/lib/python3.10/site-packages/dns/query.py", line 52, in <module>
    import httpx
  File "/home/venv/lib/python3.10/site-packages/httpx/__init__.py", line 2, in <module>
    from ._api import delete, get, head, options, patch, post, put, request, stream
  File "/home/venv/lib/python3.10/site-packages/httpx/_api.py", line 4, in <module>
    from ._client import Client
  File "/home/venv/lib/python3.10/site-packages/httpx/_client.py", line 30, in <module>
    from ._transports.default import AsyncHTTPTransport, HTTPTransport
  File "/home/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 30, in <module>
    import httpcore
  File "/home/venv/lib/python3.10/site-packages/httpcore/__init__.py", line 1, in <module>
    from ._api import request, stream
  File "/home/venv/lib/python3.10/site-packages/httpcore/_api.py", line 5, in <module>
    from ._sync.connection_pool import ConnectionPool
  File "/home/venv/lib/python3.10/site-packages/httpcore/_sync/__init__.py", line 1, in <module>
    from .connection import HTTPConnection
  File "/home/venv/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 12, in <module>
    from .._synchronization import Lock
  File "/home/venv/lib/python3.10/site-packages/httpcore/_synchronization.py", line 13, in <module>
    import trio
  File "/home/venv/lib/python3.10/site-packages/trio/__init__.py", line 22, in <module>
    from ._core import TASK_STATUS_IGNORED as TASK_STATUS_IGNORED  # isort: split
  File "/home/venv/lib/python3.10/site-packages/trio/_core/__init__.py", line 21, in <module>
    from ._local import RunVar, RunVarToken
  File "/home/venv/lib/python3.10/site-packages/trio/_core/_local.py", line 9, in <module>
    from . import _run
  File "/home/venv/lib/python3.10/site-packages/trio/_core/_run.py", line 2787, in <module>
    from ._io_epoll import (
  File "/home/venv/lib/python3.10/site-packages/trio/_core/_io_epoll.py", line 202, in <module>
    class EpollIOManager:
  File "/home/venv/lib/python3.10/site-packages/trio/_core/_io_epoll.py", line 203, in EpollIOManager
    _epoll: select.epoll = attr.ib(factory=select.epoll)
AttributeError: module 'select' has no attribute 'epoll'. Did you mean: 'poll'?
[2024-01-15 10:22:16 +0800] [31655] [INFO] Worker exiting (pid: 31655)
[2024-01-15 10:22:17 +0800] [31652] [INFO] Shutting down: Master
[2024-01-15 10:22:17 +0800] [31652] [INFO] Reason: Worker failed to boot.
    • 表现：AttributeError: module 'select' has no attribute 'epoll'. Did you mean: 'poll'?

项目介绍

  • 框架使用Flask + flask-sockets 搭建
  • 内部业务使用异步转同步，例如：asnyc、httpx等
  • 交互协议http、grpc、websocket
  • 部署：gunicorn、gevent

问题分析

  1. 部署方式分析：
    • 非Gunicorn部署（window & linux）
      from http_server.handle import app

from gevent import pywsgi
from geventwebsocket.handler import WebSocketHandler

server = pywsgi.WSGIServer(('0.0.0.0', 50003), app, handler_class=WebSocketHandler)
server.serve_forever()
可正常提供服务
epoll支持：Python官方文档
select.epoll([suzegubt=-1])
    (only supported on Linux 2.5.44 and newer.) Returns an edge polling object,
    which can be used as Edge or Level Triggered interface for I/O events;
    see section Edge and Level Trigger Polling (epoll) Objects below for the methods supported by epolling objects.
    • Gunicorn部署：如背景介绍
    • 结论：上述问题与gunicorn兼容性问题：gunicorn（gevent） <=> trio(epoll) ......> httpcore ......> httpx ......>dnspython
  2. 资源搜索分析：
    • Gunicorn（Gevent） 与 trio 存在兼容性问题：https://github.com/gevent/gevent/issues/2008
      • 无直接解决方案
    • 通过配置
      from gevent import monkey
monkey.patch_all(select=False)
    　　    or 
    export EVENTLET_NO_GREENDNS=yes
      • 对结果无效
  3. 版本兼容性处理-降级
    • 当前版本 
      • gevent                    23.9.1
gevent-websocket          0.10.1
gunicorn                  21.2.0
dnspython                 2.4.2
httpcore                  0.17.3
httpx                     0.24.1
trio                      0.24.0
trio-websocket            0.11.1
    • 经过相关包降级：
      • dsnpython => 2.3.0 无用
      • trio => 0.22.2 升降级无用
      • httpcore => 0.17.1 解决

结论：

    AttributeError: module 'select' has no attribute 'epoll' 异常原因：
  1. gunicorn部署表现：trio 与 gunicorn（gevent）的不兼容
  2. 解决办法：使用httpcore低版本 <= 0.17.12
posted @ 2024-01-15 14:48  binger0712  阅读(226)  评论(0)  编辑  收藏  举报