airflow 安装配置celery+rabbitmq celery+redis
AirFlow的安装可以参考:https://www.cnblogs.com/braveym/p/11378851.html
这里介绍的是AirFlow 安装配置celery+rabbitmq 和 celery+redis
一、airflow 安装配置celery+rabbitmq
1、安装celery和rabbitmq组件
sudo pip install apache-airflow[celery]
sudo pip install apache-airflow[rabbitmq]
2、系统安装rabbit
安装 erlang
sudo yum install erlang
安装 rabbitmq
sudo wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.6.15/rabbitmq-server-3.6.15-1.el6.noarch.rpm sudo rpm --import https://www.rabbitmq.com/rabbitmq-release-signing-key.asc sudo yum install rabbitmq-server-3.6.15-1.el6.noarch.rpm
配置rabbitmq(设置用户名root,密码rabbitmq,创建虚拟主机airflow-rabbitmq)
[bigdata@bigdata airflow]$ su Password: [root@bigdata airflow]# rabbitmqctl add_user root rabbitmq Creating user "root" [root@bigdata airflow]# rabbitmqctl add_vhost airflow-rabbitmq Creating vhost "airflow-rabbitmq" [root@bigdata airflow]# rabbitmqctl set_user_tags root airflow-rabbitmq Setting tags for user "root" to ['airflow-rabbitmq'] [root@bigdata airflow]# rabbitmqctl set_permissions -p airflow-rabbitmq root ".*" ".*" ".*" Setting permissions for user "root" in vhost "airflow-rabbitmq" [root@bigdata airflow]# rabbitmq-plugins enable rabbitmq_management The following plugins have been enabled: amqp_client cowlib cowboy rabbitmq_web_dispatch rabbitmq_management_agent rabbitmq_management Applying plugin configuration to rabbit@bigdata... started 6 plugins. [root@bigdata airflow]#
修改airflow配置
executor = CeleryExecutor
broker_url = amqp://root:rabbitmq@localhost:5672/airflow-rabbitmq
result_backend = amqp://root:rabbitmq@localhost:5672/airflow-rabbitmq
在启动worker 时,提示不能用root用户权限启动。修改方法:
echo "export C_FORCE_ROOT="True"" >> /etc/profile
source /etc/profile
二、airflow 安装配置celery+redis
安装redis
sudo yum -y install redis
修改配置文件
sudo vim /etc/redis.conf #bind 127.0.0.1 // 注释掉,使redis允许远程访问 requirepass 密码 // 修改这行,redis登录密码
开启redis服务
sudo systemctl start redis
python下安装redis库
sudo pip install redis
安装celery
sudo pip install celery
配置airflow.cfg
executor = CeleryExecutor broker_url = redis://127.0.0.1:6379/0 result_backend = redis://127.0.0.1:6379/0