django-celery启动后迅速关闭的解决办法
日志中也没有打印什么明显的错误，只是显示连接了rabbitmq后就关闭了
[2019-09-11 06:08:45,729: INFO/Beat] beat: Starting... [2019-09-11 06:08:45,731: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672// [2019-09-11 06:08:45,732: INFO/Beat] Writing entries (0)... [2019-09-11 06:08:45,740: INFO/MainProcess] mingle: searching for neighbors [2019-09-11 06:08:45,771: INFO/Beat] DatabaseScheduler: Schedule changed. [2019-09-11 06:08:45,771: INFO/Beat] Writing entries (0)... [2019-09-11 06:08:46,748: INFO/MainProcess] mingle: all alone [2019-09-11 06:08:46,749: INFO/MainProcess] beat: Shutting down... [2019-09-11 06:08:46,749: INFO/Beat] Writing entries (0)... [2019-09-11 06:08:46,749: INFO/Beat] Writing entries (0)...
查看rabbitmq的日志,这里显示rabbitmq连接成功，然后就是断开连接，（但是我还是以为是rabbitmq的问题，因为我在另外一台主机上一摸一样的环境没有问题）
=INFO REPORT==== 15-Oct-2019::02:17:54 === connection <0.305.0> (127.0.0.1:39786 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/' =INFO REPORT==== 15-Oct-2019::02:17:56 === connection <0.310.0> (127.0.0.1:39788 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/' =WARNING REPORT==== 15-Oct-2019::02:17:57 === closing AMQP connection <0.305.0> (127.0.0.1:39786 -> 127.0.0.1:5672, vhost: '/', user: 'guest'): client unexpectedly closed TCP connection =INFO REPORT==== 15-Oct-2019::02:17:58 === closing AMQP connection <0.310.0> (127.0.0.1:39788 -> 127.0.0.1:5672, vhost: '/', user: 'guest') =INFO REPORT==== 15-Oct-2019::02:17:58 === closing AMQP connection <0.302.0> (127.0.0.1:39780 -> 127.0.0.1:5672, vhost: '/', user: 'guest')
在试了重装erlang和rabbitmq等服务，折腾了一天后，第二天我换了个思路，从celery源码查看
找到Shutting down...这个日志输出的位置
def stop(self, wait=False): info('beat: Shutting down...') # 这里增加一段代码 '''''' import traceback info("start shutdown traceback ====") traceback.print_stack() '''''' self._is_shutdown.set() wait and self._is_stopped.wait() # block until shutdown done.
再次启动celery后获得以下日志
[2019-10-16 10:15:15,037: INFO/Beat] beat: Starting... [2019-10-16 10:15:15,039: INFO/Beat] Writing entries (0)... [2019-10-16 10:15:15,082: INFO/Beat] DatabaseScheduler: Schedule changed. [2019-10-16 10:15:15,082: INFO/Beat] Writing entries (0)... [2019-10-16 10:15:15,394: INFO/Beat] DatabaseScheduler: Schedule changed. [2019-10-16 10:15:15,394: INFO/Beat] Writing entries (0)... [2019-10-16 10:15:16,201: INFO/Beat] DatabaseScheduler: Schedule changed. [2019-10-16 10:15:16,201: INFO/Beat] Writing entries (0)... [2019-10-16 10:15:16,859: INFO/Beat] DatabaseScheduler: Schedule changed. [2019-10-16 10:15:16,859: INFO/Beat] Writing entries (0)... [2019-10-16 10:15:23,337: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672// [2019-10-16 10:15:28,102: INFO/Beat] Writing entries (0)... [2019-10-16 10:15:31,348: INFO/MainProcess] mingle: searching for neighbors [2019-10-16 10:15:32,357: INFO/MainProcess] mingle: all alone [2019-10-16 10:15:32,362: WARNING/MainProcess] invalid syntax (strategy.py, line 13) [2019-10-16 10:15:33,372: INFO/MainProcess] start shutdown traceback ==== [2019-10-16 10:15:33,374: WARNING/MainProcess] File "manage.py", line 15, in <module> execute_from_command_line(sys.argv) [2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/django/core/management/__init__.py" , line 381, in execute_from_command_line utility.execute() [2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/django/core/management/__init__.py" , line 375, in execute self.fetch_command(subcommand).run_from_argv(self.argv) [2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/djcelery/management/commands/celery .py", line 21, in run_from_argv ['{0[0]} {0[1]}'.format(argv)] + argv[2:], [2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline super(CeleryCommand, self).execute_from_commandline(argv))) [2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/base.py", line 311, in e xecute_from_commandline return self.handle_argv(self.prog_name, argv[1:]) [2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/celery.py", line 785, in handle_argv return self.execute(command, argv) [2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/celery.py", line 717, in execute ).run_from_argv(self.prog_name, argv[1:], command=argv[0]) [2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv return self(*args, **options) [2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/base.py", line 274, in _ _call__ ret = self.run(*args, **kwargs) [2019-10-16 10:15:33,374: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/worker.py", line 212, in run state_db=self.node_format(state_db, hostname), **kwargs [2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/worker/__init__.py", line 21 2, in start self.stop() [2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/worker/__init__.py", line 25 4, in stop self._shutdown(warm=True) [2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/worker/__init__.py", line 26 8, in _shutdown self.blueprint.stop(self, terminate=not warm) [2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bootsteps.py", line 179, in stop description=what, propagate=False, [2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bootsteps.py", line 142, in restart self.send_all(parent, method, description, propagate=propagate) [2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bootsteps.py", line 155, in send_all fun(parent, *args) [2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bootsteps.py", line 378, in stop return self.obj.stop() [2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/beat.py", line 559, in stop self.service.stop() [2019-10-16 10:15:33,375: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/beat.py", line 499, in stop traceback.print_stack() [2019-10-16 10:15:33,375: INFO/MainProcess] beat: Shutting down... [2019-10-16 10:15:33,375: INFO/Beat] Writing entries (0)... [2019-10-16 10:15:33,376: INFO/Beat] Writing entries (0)... [root@test operations]# [2019-10-16 10:15:51,178: INFO/Beat] Writing entries (0)... [2019-10-16 10:16:30,352: INFO/Beat] Writing entries (0)... [2019-10-16 10:16:46,806: INFO/Beat] Writing entries (0)... [2019-10-16 10:18:56,064: INFO/Beat] Writing entries (0)... [2019-10-16 10:19:35,231: INFO/Beat] Writing entries (0)... [2019-10-16 10:19:51,689: INFO/Beat] Writing entries (0)...
可以看出在/usr/local/python3/lib/python3.7/site-packages/celery/worker/__init__.py中的start方法中执行了stop方法，在stop中打印了beat: Shutting down...字符
def start(self): """Starts the workers main loop.""" ''' 修改start方法，去掉try,以便找到错误位置 ''' self.blueprint.start(self) #try: # self.blueprint.start(self) #except WorkerTerminate: # self.terminate() #except Exception as exc: # logger.error('Unrecoverable error: %r', exc, exc_info=True) # self.stop() #except (KeyboardInterrupt, SystemExit): # self.stop()
然后再次启动celery获得以下日志
[2019-10-16 10:19:57,241: INFO/Beat] beat: Starting... [2019-10-16 10:19:57,244: INFO/Beat] Writing entries (0)... [2019-10-16 10:19:57,287: INFO/Beat] DatabaseScheduler: Schedule changed. [2019-10-16 10:19:57,287: INFO/Beat] Writing entries (0)... [2019-10-16 10:20:00,218: INFO/Beat] DatabaseScheduler: Schedule changed. [2019-10-16 10:20:00,218: INFO/Beat] Writing entries (0)... [2019-10-16 10:20:01,023: INFO/Beat] DatabaseScheduler: Schedule changed. [2019-10-16 10:20:01,023: INFO/Beat] Writing entries (0)... [2019-10-16 10:20:01,685: INFO/Beat] DatabaseScheduler: Schedule changed. [2019-10-16 10:20:01,685: INFO/Beat] Writing entries (0)... [2019-10-16 10:20:05,548: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672// [2019-10-16 10:20:10,307: INFO/Beat] Writing entries (0)... [2019-10-16 10:20:13,561: INFO/MainProcess] mingle: searching for neighbors [2019-10-16 10:20:14,570: INFO/MainProcess] mingle: all alone [2019-10-16 10:20:14,571: WARNING/MainProcess] Traceback (most recent call last): [2019-10-16 10:20:14,571: WARNING/MainProcess] File "manage.py", line 15, in <module> [2019-10-16 10:20:14,571: WARNING/MainProcess] execute_from_command_line(sys.argv) [2019-10-16 10:20:14,571: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/django/core/management/__init__.py" , line 381, in execute_from_command_line[2019-10-16 10:20:14,572: WARNING/MainProcess] utility.execute() [2019-10-16 10:20:14,572: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/django/core/management/__init__.py" , line 375, in execute[2019-10-16 10:20:14,572: WARNING/MainProcess] self.fetch_command(subcommand).run_from_argv(self.argv) [2019-10-16 10:20:14,572: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/djcelery/management/commands/celery .py", line 21, in run_from_argv[2019-10-16 10:20:14,572: WARNING/MainProcess] ['{0[0]} {0[1]}'.format(argv)] + argv[2:], [2019-10-16 10:20:14,572: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline[2019-10-16 10:20:14,573: WARNING/MainProcess] super(CeleryCommand, self).execute_from_commandline(argv))) [2019-10-16 10:20:14,573: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/base.py", line 311, in e xecute_from_commandline[2019-10-16 10:20:14,573: WARNING/MainProcess] return self.handle_argv(self.prog_name, argv[1:]) [2019-10-16 10:20:14,573: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/celery.py", line 785, in handle_argv[2019-10-16 10:20:14,574: WARNING/MainProcess] return self.execute(command, argv) [2019-10-16 10:20:14,574: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/celery.py", line 717, in execute[2019-10-16 10:20:14,574: WARNING/MainProcess] ).run_from_argv(self.prog_name, argv[1:], command=argv[0]) [2019-10-16 10:20:14,574: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv[2019-10-16 10:20:14,574: WARNING/MainProcess] return self(*args, **options) [2019-10-16 10:20:14,575: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/base.py", line 274, in _ _call__[2019-10-16 10:20:14,575: WARNING/MainProcess] ret = self.run(*args, **kwargs) [2019-10-16 10:20:14,575: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bin/worker.py", line 212, in run[2019-10-16 10:20:14,575: WARNING/MainProcess] state_db=self.node_format(state_db, hostname), **kwargs [2019-10-16 10:20:14,575: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/worker/__init__.py", line 20 5, in start[2019-10-16 10:20:14,575: WARNING/MainProcess] self.blueprint.start(self) [2019-10-16 10:20:14,575: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bootsteps.py", line 123, in start[2019-10-16 10:20:14,576: WARNING/MainProcess] step.start(parent) [2019-10-16 10:20:14,576: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bootsteps.py", line 374, in start[2019-10-16 10:20:14,576: WARNING/MainProcess] return self.obj.start() [2019-10-16 10:20:14,576: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/worker/consumer.py", line 28 0, in start[2019-10-16 10:20:14,576: WARNING/MainProcess] blueprint.start(self) [2019-10-16 10:20:14,577: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/bootsteps.py", line 123, in start[2019-10-16 10:20:14,577: WARNING/MainProcess] step.start(parent) [2019-10-16 10:20:14,577: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/worker/consumer.py", line 64 2, in start[2019-10-16 10:20:14,577: WARNING/MainProcess] c.update_strategies() [2019-10-16 10:20:14,577: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/worker/consumer.py", line 43 9, in update_strategies[2019-10-16 10:20:14,578: WARNING/MainProcess] self.strategies[name] = task.start_strategy(self.app, self) [2019-10-16 10:20:14,578: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/app/task.py", line 448, in s tart_strategy[2019-10-16 10:20:14,578: WARNING/MainProcess] return instantiate(self.Strategy, self, app, consumer, **kwargs) [2019-10-16 10:20:14,578: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/utils/imports.py", line 53, in instantiate[2019-10-16 10:20:14,578: WARNING/MainProcess] return symbol_by_name(name)(*args, **kwargs) [2019-10-16 10:20:14,578: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/kombu/utils/__init__.py", line 96, in symbol_by_name[2019-10-16 10:20:14,578: WARNING/MainProcess] module = imp(module_name, package=package, **kwargs) [2019-10-16 10:20:14,579: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/importlib/__init__.py", line 127, in import_modul e[2019-10-16 10:20:14,579: WARNING/MainProcess] return _bootstrap._gcd_import(name[level:], package, level) [2019-10-16 10:20:14,579: WARNING/MainProcess] File "<frozen importlib._bootstrap>", line 1006, in _gcd_import [2019-10-16 10:20:14,579: WARNING/MainProcess] File "<frozen importlib._bootstrap>", line 983, in _find_and_load [2019-10-16 10:20:14,579: WARNING/MainProcess] File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked [2019-10-16 10:20:14,579: WARNING/MainProcess] File "<frozen importlib._bootstrap>", line 677, in _load_unlocked [2019-10-16 10:20:14,579: WARNING/MainProcess] File "<frozen importlib._bootstrap_external>", line 724, in exec_module [2019-10-16 10:20:14,579: WARNING/MainProcess] File "<frozen importlib._bootstrap_external>", line 860, in get_code [2019-10-16 10:20:14,580: WARNING/MainProcess] File "<frozen importlib._bootstrap_external>", line 791, in source_to_code [2019-10-16 10:20:14,580: WARNING/MainProcess] File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed [2019-10-16 10:20:14,580: WARNING/MainProcess] File "/usr/local/python3/lib/python3.7/site-packages/celery/worker/strategy.py", line 13 [2019-10-16 10:20:14,580: WARNING/MainProcess] from kombu.async.timer import to_timestamp [2019-10-16 10:20:14,580: WARNING/MainProcess] ^ [2019-10-16 10:20:14,580: WARNING/MainProcess] SyntaxError [2019-10-16 10:20:14,580: WARNING/MainProcess] : [2019-10-16 10:20:14,580: WARNING/MainProcess] invalid syntax
可以看到from kombu.async.timer import to_timestamp这句导包的时候报错 在kombu中把async目录修改成了asynchronous，同时因为python3.7以后吧async变成了一个关键字，不能作为包来引用,因此报错，所以修改async为asynchronous
再次启动celery后获得以下日志
[2019-10-16 10:27:14,497: INFO/Beat] beat: Starting... [2019-10-16 10:27:14,500: INFO/Beat] Writing entries (0)... [2019-10-16 10:27:14,545: INFO/Beat] DatabaseScheduler: Schedule changed. [2019-10-16 10:27:14,545: INFO/Beat] Writing entries (0)... [2019-10-16 10:27:14,949: INFO/Beat] DatabaseScheduler: Schedule changed. [2019-10-16 10:27:14,949: INFO/Beat] Writing entries (0)... [2019-10-16 10:27:15,029: INFO/Beat] DatabaseScheduler: Schedule changed. [2019-10-16 10:27:15,029: INFO/Beat] Writing entries (0)... [2019-10-16 10:27:15,748: INFO/Beat] DatabaseScheduler: Schedule changed. [2019-10-16 10:27:15,748: INFO/Beat] Writing entries (0)... [2019-10-16 10:27:16,427: INFO/Beat] DatabaseScheduler: Schedule changed. [2019-10-16 10:27:16,427: INFO/Beat] Writing entries (0)... [2019-10-16 10:27:22,485: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672// [2019-10-16 10:27:24,618: INFO/MainProcess] mingle: searching for neighbors [2019-10-16 10:27:25,626: INFO/MainProcess] mingle: all alone [2019-10-16 10:27:25,648: WARNING/MainProcess] /usr/local/python3/lib/python3.7/site-packages/djcelery/loaders.py:133: UserWarning: Usi ng settings.DEBUG leads to a memory leak, never use this setting in production environments! warn('Using settings.DEBUG leads to a memory leak, never ' [2019-10-16 10:27:25,648: WARNING/MainProcess] celery@test.ylyw.com ready.
当出现celery@test.ylyw.com ready.后，就可以确定celery正常运行了
