压测：celey backend为rabbitmq pk redis

使用celery的backend异步获取结果，本文使用rabbitmq 和 redis分别作为backend，代码对比如下

from celery import Celery, platforms
import time
import os
from datetime import datetime

app = Celery(‘proj‘,
             broker=‘amqp://admin:[email protected]:5672‘,     //rabbitmq
             backend=‘amqp://admin:[email protected]:5672‘,    //redis
             //backend=‘redis://ip:10013/0‘,
             include=[‘tasks‘]
             )
app.conf.update(
    CELERYD_PREFETCH_MULTIPLIER=1,
    CELERY_ACKS_LAT = True,
)

@app.task
def fun_1(n):
    return {"task_id": "0a14c9ac91de419880c4499a8f57418e",
            "status": 0,
            "desc": str(n) }

@app.task
def chk_total_callback(tasks_rets):
    end_time = datetime.now()
    print "end_time", end_time
    return 0

if __name__ == "__main__":
    app.start()

发送消息

from tasks import *
from celery import group, chain
from datetime import datetime
import time                                                            

#job_group = group([fun_1.s(i) for i in range(30000)])
beg_time = datetime.now()
for j in range(8000):
    job_group = group([fun_1.s(i) for i in range(20)])
    chain(job_group, chk_total_callback.s())()
    #time.sleep(0.25)                                                  

print "beg_time:", beg_time

压测场景一：

注：1000*20：一次发送1000次，分片20

压测场景二：

500*20: 500次，每次间隔1s，分片20

压测场景三：

8000 * 0.25 = 2000s, 任务数：8000   8000 * 20 + 8000 = 8000 * 21=168000 * 2=336000