介绍
Celery是由Python开发的一个简单、灵活、可靠的处理大量任务的分发系统，它不仅支持实时处理也支持任务调度。
- user：用户程序，用于告知celery去执行一个任务。
- broker: 存放任务（依赖RabbitMQ或Redis，进行存储）
- worker：执行任务
celery需要rabbitMQ、Redis、Amazon SQS、Zookeeper（测试中） 充当broker来进行消息的接收，并且也支持多个broker和worker来实现高可用和分布式。http://docs.celeryproject.org/en/latest/getting-started/brokers/index.html
应用场景
将响应服务器与处理服务器分开,当处理计算密集型请求时,有work来进行计算,客户端访问响应服务器时,响应服务器根据关键字去获取处理结果,不必由响应服务器进行耗时的计算操作
版本和要求
|
1
2
3
4
5
6
7
8
9
10
11
12
|
Celery version 4.0 runs on Python ❨2.7, 3.4, 3.5❩ PyPy ❨5.4, 5.5❩ This is the last version to support Python 2.7, and from the next version (Celery 5.x) Python 3.5 or newer is required. If you’re running an older version of Python, you need to be running an older version of Celery: Python 2.6: Celery series 3.1 or earlier. Python 2.5: Celery series 3.0 or earlier. Python 2.4 was Celery series 2.2 or earlier. Celery is a project with minimal funding, so we don’t support Microsoft Windows. Please don’t open any issues related to that platform. |
ps : win作为开发环境下安装3.1版本,否则会出现关于timer的异常
项目依赖
需要安装rabbitMQ、Redis. 以及python操作他们的包
快速上手
创建work
s1.py
|
1
2
3
4
5
6
7
8
9
10
11
|
import timefrom celery import Celeryapp = Celery('tasks', broker='redis://192.168.10.48:6379', backend='redis://192.168.10.48:6379')# broker 放置发布任务,redis://:password@hostname:port/db_number# backend 放置任务结果@app.taskdef xxxxxx(x, y): time.sleep(10) return x + y |
启动work
|
1
2
|
# 执行 s1.py 创建worker（终端执行命令）：celery worker -A s1 -l info # -A 文件名 |
发布一个任务
|
1
2
3
4
5
6
7
8
9
|
#!/usr/bin/env python# -*- coding:utf-8 -*-from s1 import xxxxxx# 立即告知celery去执行xxxxxx任务，并传入两个参数result = xxxxxx.delay(4, 4)print(result.id)# 执行此脚本即可将任务发布,会有一个work接收并执行xxxxxx, |
获取执行结果
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
from celery.result import AsyncResultfrom s1 import appasync = AsyncResult(id="f0b41e83-99cf-469f-9eff-74c8dd600002", app=app)# AsyncResult(id="上面的resutl.id", app=Celery实例)if async.successful(): # 判断是否执行完毕 result = async.get() # async.get()是阻塞的,可以设置超时时间 print(result) # result.forget() # 将结果删除elif async.failed(): print('执行失败')elif async.status == 'PENDING': print('任务等待中被执行')elif async.status == 'RETRY': print('任务异常后正在重试')elif async.status == 'STARTED': print('任务已经开始被执行') |
多实例应用同一方法
|
1
2
3
4
5
6
7
8
9
|
from celery import Celeryfrom celery import shared_taskapp1 = Celery('tasks1', broker='redis://127.0.0.1:6379', backend='redis://127.0.0.1:6379')app2 = Celery('tasks2', broker='redis://127.0.0.1:6379', backend='redis://127.0.0.1:6379')@shared_taskdef xxxxxx(x, y): return x + y |
如果两个app的某个任务执行的是相同的操作,可以使用上面的方式. 该文件下的所有Celery对象都可调用xxxxxx方法,如果在该文件下为每个Celery实例注册(不同逻辑)任务,因为在同一名称空间内,会出现一些问题
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
|
# workfrom celery import Celeryfrom celery import shared_taskapp1 = Celery('tasks1', broker='redis://127.0.0.1:6379', backend='redis://127.0.0.1:6379')app2 = Celery('tasks2', broker='redis://127.0.0.1:6379', backend='redis://127.0.0.1:6379')@app1.taskdef xxxxxx(x, y): return x + y@app2.taskdef xxxxxx(x, y): return x - y# 发布任务from s1 import xxxxxximport datetime# 构建一个utc时间对象ctime = datetime.datetime.now()utc_ctime = datetime.datetime.utcfromtimestamp(ctime.timestamp())ctime_x = utc_ctime + datetime.timedelta(seconds=1)result = xxxxxx.apply_async(args=[1, 3], eta=ctime_x)print(type(result))print(result.id)# 获取到的结果app1 : 执行完成并获取结果： -2app2 : 执行完成并获取结果： -2 |
多任务结构
|
1
2
3
4
5
6
|
pro_cel ├── celery_tasks# celery相关文件夹 │ ├── celery.py # celery连接和配置相关文件,必须叫这个名 │ └── tasks.py # 所有任务函数 ├── check_result.py # 检查结果 └── send_task.py # 触发任务 |
pro_cel/celery_tasks/celery
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
#!/usr/bin/env python# -*- coding:utf-8 -*-from celery import Celerycelery = Celery('xxxxxx', broker='redis://192.168.0.111:6379', backend='redis://192.168.0.111:6379', include=['celery_tasks.tasks']) # 函数位置# 时区celery.conf.timezone = 'Asia/Shanghai'# 是否使用UTCcelery.conf.enable_utc = False |
pro_cel/celery_tasks/tasks.py
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
#!/usr/bin/env python# -*- coding:utf-8 -*-import timefrom .celery import celery@celery.taskdef xxxxx(*args, **kwargs): time.sleep(5) return "任务结果"@celery.taskdef hhhhhh(*args, **kwargs): time.sleep(5) return "任务结果" |
pro_cel/check_result.py
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
#!/usr/bin/env python# -*- coding:utf-8 -*-from celery.result import AsyncResultfrom celery_tasks.celery import celeryasync = AsyncResult(id="ed88fa52-11ea-4873-b883-b6e0f00f3ef3", app=celery)if async.successful(): result = async.get() print(result) # result.forget() # 将结果删除elif async.failed(): print('执行失败')elif async.status == 'PENDING': print('任务等待中被执行')elif async.status == 'RETRY': print('任务异常后正在重试')elif async.status == 'STARTED': print('任务已经开始被执行') |
pro_cel/check_result.py
|
1
2
3
4
5
6
7
8
|
#!/usr/bin/env python# -*- coding:utf-8 -*-import celery_tasks.tasks# 立即告知celery去执行xxxxxx任务，并传入两个参数result = celery_tasks.tasks.xxxxx.delay(4, 4)print(result.id) |
启动work
|
1
|
celery worker -A celery_tasks -l info |
定时任务
设定时间让celery执行一个任务(需是utc时间)
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
from s1 import xxxxxx# 告知celery，什么时候帮我执行任务import datetime# ctime = datetime.datetime(year=2019,month=2,day=8)# 构建一个utc时间对象ctime = datetime.datetime.now()utc_ctime = datetime.datetime.utcfromtimestamp(ctime.timestamp())ctime_x = utc_ctime + datetime.timedelta(seconds=1)result = xxxxxx.apply_async(args=[1, 3], eta=ctime_x)print(type(result))print(result.id) |
类似于contab的定时任务
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
from celery import Celeryfrom celery.schedules import crontab app = Celery('tasks', broker='amqp://47.98.134.86:5672', backend='amqp://47.98.134.86:5672', include=['proj.s1', ])app.conf.timezone = 'Asia/Shanghai'app.conf.enable_utc = False app.conf.beat_schedule = { # 'add-every-10-seconds': { # 'task': 'proj.s1.add1', # 执行方法 # 'schedule': 10.0, # 时间设置/10秒 # 'args': (16, 16) # 方法参数 # }, 'add-every-12-seconds': { 'task': 'proj.s1.add1', 'schedule': crontab(minute=42, hour=8, day_of_month=11, month_of_year=4), # 哪月哪天哪时哪分执行 'args': (16, 16) },} |
