玩了一个星期的clannad，是时候干点事了。
折腾了下celery周期性任务：
celery提供了一个叫celery beat的服务，用于定时驱使worker执行任务。也就是说，如果本地没有活动的worker，它将不会得到任何执行结果，他只是负责把任务消息传到rabbitmq，一旦启动一个可用worker，则自动从rabbitmq获取任务信息并执行。
与此配置相关的参数是CELERYBEAT_SCHEDULE，我把我的celery应用proj的所有配置内容都放置在一个config.py文件中：
from __future__ import absolute_import
from datetime import timedelta
from celery.schedules import crontab
CELERY_TASK_RESULT_EXPIRES=3600
CELERY_TASK_SERIALIZER=‘json‘
CELERY_ACCEPT_CONTENT=[‘json‘]
CELERY_RESULT_SERIALIZER=‘json‘
CELERYBEAT_SCHEDULE = {
‘add-every-1-min‘: {
‘task‘: ‘proj.agent.add‘,
‘schedule‘: crontab(),
‘args‘: (16, 16),
},
}
#CELERYBEAT_SCHEDULE = {
# ‘add-every-2-seconds‘: {
# ‘task‘: ‘proj.agent.add‘,
# ‘schedule‘: timedelta(seconds=3),
# ‘args‘: (16, 16)
# },
#}
CELERY_TIMEZONE = ‘UTC‘
目前的定时任务是：
add-every-4-s
task指定了相应的任务：proj目录下agent模块的add函数
schedule指定了定时工具，这里是celery.schedules的crontab
args是任务的参数
此时我们回到proj所在的目录中，启动一个worker：
[email protected]:~/celeryapp/configtest# ls celerybeat-schedule logging proj
[email protected]:~/celeryapp/configtest# celery -A proj worker --loglevel=INFO /usr/local/lib/python2.7/dist-packages/celery/platforms.py:766: RuntimeWarning: You are running the worker with superuser privileges, which is absolutely not recommended! Please specify a different user using the -u option. User information: uid=0 euid=0 gid=0 egid=0 uid=uid, euid=euid, gid=gid, egid=egid, -------------- [email protected]roup0.hzg.com v3.1.17 (Cipater) ---- **** ----- --- * *** * -- Linux-3.13.0-24-generic-x86_64-with-Ubuntu-14.04-trusty -- * - **** --- - ** ---------- [config] - ** ---------- .> app: proj:0x7f0027635510 - ** ---------- .> transport: amqp://guest:**@localhost:5672// - ** ---------- .> results: amqp://[email protected]// - *** --- * --- .> concurrency: 1 (prefork) -- ******* ---- --- ***** ----- [queues] -------------- .> celery exchange=celery(direct) key=celery [tasks] . proj.agent.add . proj.agent.mul . proj.agent.writefile . proj.agent.xsum [2015-05-24 15:00:37,873: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672// [2015-05-24 15:00:37,940: INFO/MainProcess] mingle: searching for neighbors [2015-05-24 15:00:38,980: INFO/MainProcess] mingle: all alone [2015-05-24 15:00:39,021: WARNING/MainProcess] [email protected] ready.
worker启动成功，此时再开一个终端，启动beat服务：
[email protected]:~/celeryapp/configtest# celery -A proj beat -s celerybeat-schedule celery beat v3.1.17 (Cipater) is starting. __ - ... __ - _ Configuration -> . broker -> amqp://guest:**@localhost:5672// . loader -> celery.loaders.app.AppLoader . scheduler -> celery.beat.PersistentScheduler . db -> celerybeat-schedule . logfile -> [stderr]@%INFO . maxinterval -> now (0s) [2015-05-24 15:02:53,761: INFO/MainProcess] beat: Starting... [2015-05-24 15:03:00,000: INFO/MainProcess] Scheduler: Sending due task add-every-1-min (proj.agent.add) #已经相隔1min了 [2015-05-24 15:04:00,066: INFO/MainProcess] Scheduler: Sending due task add-every-1-min (proj.agent.add)
返回看看worker的输出：
[2015-05-24 15:01:50,827: INFO/MainProcess] Task proj.agent.add[9b6f962a-9b66-4fde-916f-fc5a951ad599] succeeded in 0.0342152439989s: {‘value‘: ‘32‘}
[2015-05-24 15:02:24,923: INFO/MainProcess] Received task: proj.agent.add[e4b9840b-09f6-4db6-88c1-2a418b11d393]
[2015-05-24 15:02:24,947: INFO/MainProcess] Task proj.agent.add[e4b9840b-09f6-4db6-88c1-2a418b11d393] succeeded in 0.0200459280004s: {‘value‘: ‘32‘}
[2015-05-24 15:03:00,015: INFO/MainProcess] Received task: proj.agent.add[98f44dd1-e6e2-4457-bfd6-ff59d0ee6d2f]
[2015-05-24 15:03:00,031: INFO/MainProcess] Task proj.agent.add[98f44dd1-e6e2-4457-bfd6-ff59d0ee6d2f] succeeded in 0.0125673500006s: {‘value‘: ‘32‘}
这就是周期性任务的执行。
遇到的坑：
在配置文件中，from __future__ import absolute_import这一行很关键，如果没有这一行，
from celery.schedules import crontab
这个命令执行时会报错，celery beat无法正常启动。
补充：
默认情况下，celery beat使用UTC时区，你也可以配置其他时区：
CELERY_TIMEZONE = ‘Europe/London‘
关于设置任务执行周期，你可以通过datetime的timedelta设置，可以让任务执行间隔精确到秒，相应的配置如下：
CELERYBEAT_SCHEDULE = {
‘add-every-2-seconds‘: {
‘task‘: ‘proj.agent.add‘,
‘schedule‘: timedelta(seconds=3),
‘args‘: (16, 16)
},
}
也可以用crontab风格的：
CELERYBEAT_SCHEDULE = {
# Executes every Monday morning at 7:30 A.M
‘add-every-1-min‘: {
‘task‘: ‘proj.agent.add‘,
‘schedule‘: crontab(),
‘args‘: (16, 16),
},
}
关于一个CELERYBEAT_SCHEDULE的可以配置的参数，以及crontab的详细示例，请参见celery官方文档。
关于启动celery beat的tips，我这里只贴原文：
Starting the Scheduler
To start the celery beat service:
$ celery -A proj beatYou can also start embed beat inside the worker by enabling workers -B option, this is convenient if you will never run more than one worker node, but it’s not commonly used and for that reason is not recommended for production use:
$ celery -A proj worker -BBeat needs to store the last run times of the tasks in a local database file (namedcelerybeat-schedule by default), so it needs access to write in the current directory, or alternatively you can specify a custom location for this file:
$ celery -A proj beat -s /home/celery/var/run/celerybeat-schedule