安装airflow
1. 设置环境变量
echo "export AIRFLOW_HOME=/opt/airflow" >> ~/.bashrc
source ~/.bashrc
2.pip install apache-airflow==1.10.5
3.修改airflow.cfg 配置
注释默认的DB 配置,设置为mysql DB url,在MySQL中 新建数据库 airflow
#executor = SequentialExecutor
executor = LocalExecutor
#sql_alchemy_conn = sqlite:////opt/airflow/airflow.db
sql_alchemy_conn = mysql://root:123456@localhost:3306/airflow
4. airflow initdb
如果报错 ModuleNotFoundError: No module named 'MySQLdb'
[2019-10-12 12:18:41,600] {settings.py:213} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=5466
Traceback (most recent call last):
File "/opt/anaconda3/bin/airflow", line 21, in <module>
from airflow import configuration
File "/opt/anaconda3/lib/python3.7/site-packages/airflow/__init__.py", line 44, in <module>
settings.initialize()
File "/opt/anaconda3/lib/python3.7/site-packages/airflow/settings.py", line 338, in initialize
configure_orm()
File "/opt/anaconda3/lib/python3.7/site-packages/airflow/settings.py", line 225, in configure_orm
engine = create_engine(SQL_ALCHEMY_CONN, **engine_args)
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/__init__.py", line 435, in create_engine
return strategy.create(*args, **kwargs)
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 87, in create
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/opt/anaconda3/lib/python3.7/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 118, in dbapi
return __import__("MySQLdb")
ModuleNotFoundError: No module named 'MySQLdb'
解决方法:
sudo apt-get install aptitude
sudo aptitude install libmysqlclient-dev
pip install mysqlclient
Noted: **sudo apt-get install aptitude ** and sudo aptitude install libmysqlclient-dev 是用于解决
ERROR: Complete output from command python setup.py egg_info:
ERROR: /bin/sh: 1: mysql_config: not found
/bin/sh: 1: mariadb_config: not found
/bin/sh: 1: mysql_config: not found
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-1xi26n1k/mysqlclient/setup.py", line 16, in <module>
metadata, options = get_config()
File "/tmp/pip-install-1xi26n1k/mysqlclient/setup_posix.py", line 61, in get_config
libs = mysql_config("libs")
File "/tmp/pip-install-1xi26n1k/mysqlclient/setup_posix.py", line 29, in mysql_config
raise EnvironmentError("%s not found" % (_mysql_config_path,))
OSError: mysql_config not found
----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-1xi26n1k/mysqlclient/
5.启动服务
airflow webserver -p 9888
airflow scheduler