celery 相关问题

一、 celery SQS 的 pycurl 问题

celery 利用 AWS SQS 安装 pycurl 时会出现 “ImportError: pycurl” 相关的错误，解决方法如下：

先把原先安装的 pycurl 删除
重新安装 pycurl，安装步骤如下：

安装命令：

# 1. Exporting two constants (as stated here pyca/cryptography#3489):

export CPPFLAGS=-I/usr/local/opt/openssl/include
export LDFLAGS=-L/usr/local/opt/openssl/lib


# Installing with pip

pip install pycurl --global-option="--with-openssl"

参考链接：

https://github.com/transloadit/python-sdk/issues/4#issuecomment-347009356

二、 celery 任务没有被注册的问题

问题大致如下：

[2020-09-30 11:04:45,917: ERROR/MainProcess] Received unregistered task of type 'tasks.xxx.xxxxxxxxx'.
The message has been ignored and discarded.

Did you remember to import the module containing this task?
Or maybe you're using relative imports?

Please see
http://docs.celeryq.org/en/latest/internals/protocol.html
for more information.

The full contents of the message body was:
b'xxxxxxxx' (367b)
Traceback (most recent call last):
  File "/xxxxxx/venv/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 555, in on_task_received
    strategy = strategies[type_]
KeyError: 'tasks.xxx.xxxxxxxxx'

解决方法：

在创建 celery 的时候加上 include=['tasks.asyn_tasks'] ，列表中的内容表示任务的模块的路径

from celery import Celery


def make_celery(app):
    celery = Celery(
        app.import_name,
        backend=app.config['CELERY_RESULT_BACKEND'],
        broker=app.config['CELERY_BROKER_URL'],
        include=['tasks.asyn_tasks']    # 加上 include 任务路径
    )
    celery.conf.update(app.config)

    class ContextTask(celery.Task):
        def __call__(self, *args, **kwargs):
            with app.app_context():
                return self.run(*args, **kwargs)

    celery.Task = ContextTask
    return celery