airflow worker 出现 (2006, 'MySQL server has gone away')以及raise AirflowException('Celery command failed')airflow 设置分布式部署主从worker不能协调运行

OperationalError: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away')
[SQL: SELECT celery_taskmeta.id AS celery_taskmeta_id, celery_taskmeta.task_id AS celery_taskmeta_task_id, celery_taskmeta.status

重启能解决上述问题

[2020-04-20 17:25:40,165: ERROR/ForkPoolWorker-1] None
[2020-04-20 17:25:40,172: ERROR/ForkPoolWorker-1] Task airflow.executors.celery_executor.execute_command[5253e28b-4219-42bc-81ca-2b8951a7123e] raised unexpected: AirflowException('Celery command failed',)
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python2.7/site-packages/celery/app/trace.py", line 385, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/ubuntu/.local/lib/python2.7/site-packages/celery/app/trace.py", line 650, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/airflow/executors/celery_executor.py", line 72, in execute_command
raise AirflowException('Celery command failed')
AirflowException: Celery command failed

 

上述问题，未找到对应的执行程序，认真检查执行程序是否在所在目录；另外确保slave worker能执行master程序请设置如下操作:

sudo airflow worker -cn=celery@hosttname -p -D

 

master schduler -p