Airflow
Airflow
文档地址:https://airflow.apache.org/docs/apache-airflow/stable/index.html
动态任务
def taskList(task_conf,batch_size):
task = BashOperator(
task_id='',
bash_command=bash_command,
dag=main_dag
)
# 4 返回任务
return task
list_task = []
for line in lines:
list_task.append(taskList(line, batch_size))
start > list_task > end
命令
##所有任务置成功
airflow backfill **_dag -s 2022-01-01 -e 2022-05-12 -i -m --reset_dagruns -y -sd path
##单运行单个节点
airflow backfill **_dag -s 2022-01-01 -e 2022-05-12 -i --reset_dagruns -y -sd path -t hive2db_task
自定义日期
def date_add(ds_nodash ,days):
execution_date = datetime.strptime(ds_nodash ,'%Y%m%d')
new_date = (execution_date + timedelta(days=days)).strftime('%Y%m%d')
return str(new_date)
main_dag = DAG(
dag_id=dag_id,
default_args=default_args,
description=dag_description,
catchup=False,
schedule_interval=schedule_interval,
user_defined_macros={
"date_add": date_add,
}
)
'{{ date_add2(ds ,-2) }}'
搬砖多年终不得要领,遂载源码看之望得真经。