Flink学习笔记——Flink on YARN

Flink集群部署的方式有以下几种，在本文中主要介绍Flink on yarn：

Yarn
Mesos
Docker/Kubernetes
Standalone

 

参考：

https://www.slideshare.net/tillrohrmann/redesigning-apache-flinks-distributed-architecture-flink-forward-2017

 

Flink on yarn有2种运行方式：

 

Flink架构

 

 

第一种是Session mode

定义参考：《Stream Processing with Apache Flink: Fundamentals, Implementation, and Operation of Streaming Applications》

https://books.google.com.sg/books?id=7OiRDwAAQBAJ&pg=PT265&lpg=PT265&dq=flink+job+mode+session+mode&source=bl&ots=0fMg-gnzyf&sig=ACfU3U1-W4Az3WWJiUBwTpm0tGbsi2UpGw&hl=zh-CN&sa=X&ved=2ahUKEwiMwJjS-uDpAhVTIqYKHbgABkwQ6AEwBnoECAoQAQ#v=onepage&q=flink%20job%20mode%20session%20mode&f=false

 

 

 

即启动一个yarn session，然后submit flink job到这个yarn session上

有2个步骤：

1.启动一个yarn session

 

 

2.提交任务到flink集群上

 

参考：

https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html

 

第二种是Job mode

即提交一个独立的flink任务到yarn上面，一旦这个job结束了，Flink集群上的资源就会释放

 

flink kerberos参考

https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/security/security-kerberos.html

即在conf/flink-conf.yaml中配置

security.kerberos.login.use-ticket-cache: false
security.kerberos.login.keytab: /path/xxx.keytab
security.kerberos.login.principal: xxx
security.kerberos.login.contexts: Client,KafkaClient