airflow分布式部署（一）Python环境

一、Python环境安装

airflow依赖Python3，所以安装之前需要保证服务器有Python环境

#安装Python3
#安装依赖环境
yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel
#下载安装包
wget https://www.python.org/ftp/python/3.6.8/Python-3.6.8.tgz
#创建python3目录
mkdir -p /usr/local/python3
#解压缩
tar -zxvf Python-3.6.8.tgz
#编译安装,需要c环境
yum install gcc
cd Python-3.6.8
./configure --prefix=/usr/local/python3
make
make install
#建立python3的软链
ln -s /usr/local/python3/bin/python3 /usr/bin/python3
ln -s /usr/local/python3/bin/pip3 /usr/bin/pip3
#将/usr/local/python3/bin加入PATH
...
PATH=$PATH:$HOME/bin:/usr/local/python3/bin
export PATH
...
source ~/.bash_profile
#检查是否生效
python3 -V
pip3 -V

 

 

#systemctl自启动报错解决
#Q:No such file or directory: 'airflow': 'airflow'
#A:将python3的路径加入到PATH：which python3

export PATH=$PATH:/root/miniconda3/envs/py36/bin/
source /etc/profile

二、依赖包安装

airflow依赖大量第三方包，如果都一一安装较为困难，本次采用安装miniconda包，可以达到事半功倍的效果

wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh

bash Miniconda3-latest-Linux-x86_64.sh

source ~/.bashrc

conda create -n py36 python=3.6 anaconda

conda activate py36

安装完miniconda之后，就可以安装airflow的包了

pip install 'apache-airflow[celery]'
pip install 'apache-airflow[redis]'
pip install 'apache-airflow[mysql]'