airflow学习笔记——operator

operator用于产生特定的DAG节点

https://airflow.apache.org/docs/apache-airflow/stable/python-api-ref.html#operators

 

下面是常用的operator及其用法

BaseOperator

https://airflow.apache.org/docs/apache-airflow/stable/_modules/airflow/models/baseoperator.html

用法：

 

BaseSensorOperator

https://airflow.apache.org/docs/apache-airflow/stable/_modules/airflow/sensors/base.html#BaseSensorOperator.poke

用法：

 

MySQLToS3Operator

这个已经过时了，建议使用SqlToS3Operator

https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_modules/airflow/providers/amazon/aws/transfers/mysql_to_s3.html

用法：

 

SqlToS3Operator

https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/transfer/sql_to_s3.html#howto-operator-sqltos3operator

用法：

 

MongoToS3Operator

https://airflow.apache.org/docs/apache-airflow-providers-amazon/1.2.0/_modules/airflow/providers/amazon/aws/transfers/mongo_to_s3.html

用法：

 

BashOperator：执行bash命令

https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/bash.html

 

PythonOperator：调用python代码

https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html

 

SparkOperators

https://airflow.apache.org/docs/apache-airflow-providers-apache-spark/stable/operators.html

 

EmailOperator：发送邮件

https://bhavaniravi.com/blog/sending-emails-from-airflow/

 

HTTPOperator : 发送 HTTP 请求

https://airflow.apache.org/docs/apache-airflow-providers-http/stable/operators.html

 

MySqlOperator : 执行 SQL 命令

https://airflow.apache.org/docs/apache-airflow-providers-mysql/stable/operators.html

 

SSHExecuteOperator，airflow1.9.0之后废除，改成 SSHOperator

https://newbedev.com/airflow-how-to-ssh-and-run-bashoperator-from-a-different-server

 

SparkSubmitOperator：提交spark任务

https://airflow.apache.org/docs/apache-airflow-providers-apache-spark/stable/_api/airflow/providers/apache/spark/operators/spark_submit/index.html#airflow.providers.apache.spark.operators.spark_submit.SparkSubmitOperator

　　

除了这些基本的构建块之外，还有更多的特定处理器：DockerOperator，HiveOperator，S3FileTransferOperator，PrestoToMysqlOperator，SlackOperator