celery signature app instance 创建简单说明

celery 对于celery 实例的处理比较有意思，没有的会创建默认的，如果定义了会基于python 的threading.local 进行引用，所以我们会看到

celery 一些代码使用上比较有意思（group，chain等，不需要明确指定app 就可以进行任务的创建以及数据获取）

一个简单通过signature进行group 任务的处理

• 示例

from celery import Celery,group,signature
app = Celery('app',backend="redis://localhost:6379/0", broker='redis://localhost:6379/0')
print("define app: ",app)

taska = app.signature('app.task_add',args=(2, 3),options={'queue': 'queue1'})
taskb = app.signature('app.task_subtract',args=(4, 5),options={'queue': 'queue2'})

group_tasks = group(taska, taskb).apply_async()
print(group_tasks.get())

• 相似效果

from celery import Celery,group,signature
app = Celery('app',backend="redis://localhost:6379/0", broker='redis://localhost:6379/0')
print("define app: ",app)

taska = signature('app.task_add',args=(2, 3),options={'queue': 'queue1'})

print("taska app:", taska.app)
taskb = signature('app.task_subtract',args=(4, 5),options={'queue': 'queue2'})

group_tasks = group(taska, taskb).apply_async()
print(group_tasks.get())

内部处理简单说明

实际上核心还是当执行的时候，定义只是一个声明，在具体执行的时候会进行celery instance 的选择以及创建

• group apply_async 简单处理

def apply_async(self, args=None, kwargs=None, add_to_parent=True,
                producer=None, link=None, link_error=None, **options):
    args = args if args else ()
    if link is not None:
        raise TypeError('Cannot add link to group: use a chord')
    if link_error is not None:
        raise TypeError(
            'Cannot add link to group: do that on individual tasks')
    # 此处app 是一个属性
    app = self.app
    if app.conf.task_always_eager:
        return self.apply(args, kwargs, **options)
    if not self.tasks:
        return self.freeze()

    options, group_id, root_id = self._freeze_gid(options)
    tasks = self._prepared(self.tasks, [], group_id, root_id, app)
    p = barrier()
    results = list(self._apply_tasks(tasks, producer, app, p,
                                     args=args, kwargs=kwargs, **options))
    result = self.app.GroupResult(group_id, results, ready_barrier=p)
    p.finalize()

    # - Special case of group(A.s() | group(B.s(), C.s()))
    # That is, group with single item that's a chain but the
    # last task in that chain is a group.
    #
    # We cannot actually support arbitrary GroupResults in chains,
    # but this special case we can.
    if len(result) == 1 and isinstance(result[0], GroupResult):
        result = result[0]

    parent_task = app.current_worker_task
    if add_to_parent and parent_task:
        parent_task.add_trail(result)
    return result   

app 的处理

@property
def app(self):
    app = self._app
    if app is None:
        try:
            # 为空，先尝试基于task获取
            app = self.tasks[0].app
        except LookupError:
            pass
    return app if app is not None else current_app  # 否则通过current_app

current_app 的处理(包含了_get_current_app 以及_set_current_app)

def _get_current_app():
    if default_app is None:
        #: creates the global fallback app instance.
        from celery.app.base import Celery
        set_default_app(Celery(
            'default', fixups=[], set_as_current=False,
            loader=os.environ.get('CELERY_LOADER') or 'default',
        ))
    return _tls.current_app or default_app


def _set_current_app(app):
    _tls.current_app = app

_set_current_app 的处理

实际上当我们实例化celery 的时候会使用到此方法

# 此方法有一个参数 celery的set_as_current 为true执行，默认这个参数为True
def set_current(self):
    """Make this the current app for this thread."""
    _set_current_app(self)

celery 定义

def __init__(self, main=None, loader=None, backend=None,
             amqp=None, events=None, log=None, control=None,
             set_as_current=True, tasks=None, broker=None, include=None,
             changes=None, config_source=None, fixups=None, task_cls=None,
             autofinalize=True, namespace=None, strict_typing=True,
             **kwargs):

说明

celery 一些使用有时让我们会比较迷惑，实际上是内部隐藏了比较多的东西，结合源码学习会加深对于celery 的使用

参考资料

https://docs.celeryq.dev/en/stable/userguide/canvas.html