|NO.Z.00065|——————————|BigDataEnd|——|Hadoop&PB级数仓.V07|——|PB数仓.v07|调度系统Airflow|核心交易|调度任务集成|

一、核心交易调度脚本的角度顺序

### --- 核心交易分析
~~~     depends_on_past ，设置为True时，上一次调度成功了，才可以触发。

# 加载ODS数据（DataX迁移数据）
/data/yanqidw/script/trade/ods_load_trade.sh

# 加载DIM层数据
/data/yanqidw/script/trade/dim_load_product_cat.sh
/data/yanqidw/script/trade/dim_load_shop_org.sh
/data/yanqidw/script/trade/dim_load_payment.sh
/data/yanqidw/script/trade/dim_load_product_info.sh

# 加载DWD层数据
/data/yanqidw/script/trade/dwd_load_trade_orders.sh

# 加载DWS层数据
/data/yanqidw/script/trade/dws_load_trade_orders.sh

# 加载ADS层数据
/data/yanqidw/script/trade/ads_load_trade_order_analysis.sh

二、核心交易调度任务集成

### --- 创建核心交易调度任务集成脚本

(env) [root@hadoop02 ~]# vim $AIRFLOW_HOME/dags/coretrade.py

from datetime import timedelta

import datetime
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.utils.dates import days_ago

# 定义dag的缺省参数
default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': '2020-06-20',
    'email': ['yanqi_vip@yeah.net'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

# 定义DAG
coretradedag = DAG(
    'coretrade',
    default_args=default_args,
    description='core trade analyze',
    schedule_interval='30 0 * * *',
)

today=datetime.date.today()
oneday=timedelta(days=1)
yesterday=(today-oneday).strftime("%Y-%m-%d")

odstask = BashOperator(
    task_id='ods_load_data',
    depends_on_past=False,
    bash_command='sh /data/yanqidw/script/trade/ods_load_trade.sh ' + yesterday,
    dag=coretradedag
)

dimtask1 = BashOperator(
    task_id='dimtask_product_cat',
    depends_on_past=False,
    bash_command='sh /data/yanqidw/script/trade/dim_load_product_cat.sh ' + yesterday,
    dag=coretradedag
)

dimtask2 = BashOperator(
    task_id='dimtask_shop_org',
    depends_on_past=False,
    bash_command='sh /data/yanqidw/script/trade/dim_load_shop_org.sh ' + yesterday,
    dag=coretradedag
)

dimtask3 = BashOperator(
    task_id='dimtask_payment',
    depends_on_past=False,
    bash_command='sh /data/yanqidw/script/trade/dim_load_payment.sh ' + yesterday,
    dag=coretradedag
)

dimtask4 = BashOperator(
    task_id='dimtask_product_info',
    depends_on_past=False,
    bash_command='sh /data/yanqidw/script/trade/dim_load_product_info.sh ' + yesterday,
    dag=coretradedag
)

dwdtask = BashOperator(
    task_id='dwd_load_data',
    depends_on_past=False,
    bash_command='sh /data/yanqidw/script/trade/dwd_load_trade_orders.sh '+ yesterday,
    dag=coretradedag
)

dwstask = BashOperator(
    task_id='dws_load_data',
    depends_on_past=False,
    bash_command='sh /data/yanqidw/script/trade/dws_load_trade_orders.sh ' + yesterday,
    dag=coretradedag
)

adstask = BashOperator(
    task_id='ads_load_data',
    depends_on_past=False,
    bash_command='sh /data/yanqidw/script/trade/ads_load_trade_order_analysis.sh ' + yesterday,
    dag=coretradedag
)

odstask >> dimtask1
odstask >> dimtask2
odstask >> dimtask3
odstask >> dimtask4
odstask >> dwdtask

dimtask1 >> dwstask
dimtask2 >> dwstask
dimtask3 >> dwstask
dimtask4 >> dwstask
dwdtask >> dwstask

dwstask >> adstask

### --- 执行coretrade.py案例文件

~~~     # 执行命令检查脚本是否有错误。如果命令行没有报错，就表示没问题
(env) [root@hadoop02 ~]# python $AIRFLOW_HOME/dags/coretrade.py

~~~     # 查看生效的 dags

(env) [root@hadoop02 ~]# airflow list_dags -sd $AIRFLOW_HOME/dags
~~~输出参数
-------------------------------------------------------------------
DAGS
-------------------------------------------------------------------
HelloWorldDag
coretrade

~~~     # 查看指定dag中的task

(env) [root@hadoop02 ~]# airflow list_tasks coretrade
~~~输出参数
ads_load_data
dimtask_payment
dimtask_product_cat
dimtask_product_info
dimtask_shop_org
dwd_load_data
dws_load_data
ods_load_data     

~~~     # 测试dag中的调用脚本

(env) [root@hadoop02 ~]# airflow load_ods coretrade ods_load_data 20211010

三、通过airflow.UI查看新建的DAGS任务：查看coretrade的定时任务

### --- 查看定时任务脚本调用流程

airflow list_dags
airflow list_tasks coretrade --tree

---

 

 

 

 

 

 

 

 

 

---

Walter Savage Landor:strove with none,for none was worth my strife.Nature I loved and, next to Nature, Art:I warm'd both hands before the fire of life.It sinks, and I am ready to depart

                                                                                                                                                   ——W.S.Landor

---