Flink 1.10 on yarn集群搭建(hadoop 3.1.3)
2020双十 阿里云服务器ECS团购特惠链接
前提条件：需先安装zookeeper与hadoop
zookeeper安装
hadoop安装
一、系统配置
3台虚拟机
hadoop1: 4g内存 2核 80g硬盘
hadoop2 1g内存 1核 8g硬盘
hadoop3: 1g内存 1核 8g硬盘
二、基础概念
1、Flink Client
2、JobManager
3、TaskManager
三、flink集群搭建(on yarn)
集群搭建的主要内容为JobManager高可用配置
1、下载与解压
(1)下载安装包flink-1.10.0-bin-scala_2.12.tgz至hadoop1节点
curl -O http://apache.mirrors.hoobly.com/flink/flink-1.10.0/flink-1.10.0-bin-scala_2.12.tgz
(2)远程传输到hadoop2、hadoop3节点
scp flink-1.10.0-bin-scala_2.12.tgz root@hadoop2:`pwd`
scp flink-1.10.0-bin-scala_2.12.tgz root@hadoop3:`pwd`
(3)解压安装包
tar zxvf flink-1.10.0-bin-scala_2.12.tgz
安装结束
2、JobManager高可用配置
(1)yarn-site.xml
文件目录：/usr/local/softwareinstall/hadoop-3.1.3/etc/hadoop/yarn-site.xml(hadoop安装目录下的yarn集群配置文件）
添加如下配置
<property>
<name>yarn.resourcemanager.am.max-attempts</name>
<value>4</value>
<description>
The maximum number of application master execution attempts.
</description>
</property>
(2)flink配置文件flink-conf.yaml
文件目录：(/usr/local/softwareinstall/flink-1.10.0/conf/flink-conf.yaml)
高可用设置
high-availability: zookeeper
high-availability.zookeeper.quorum: hadoop1:2181,hadoop2:2181,hadoop3:2181
high-availability.storageDir: hdfs:///flink/recovery
high-availability.zookeeper.path.root: /flink
yarn.application-attempts: 10
JobManager与TaskManager配置
(3)添加flink环境变量/etc/profile
vim /etc/profile
#flink环境变量配置
export FLINK_HOME=/usr/local/softwareinstall/flink-1.10.0
export PATH=$PATH:$FLINK_HOME/bin
#flink on yarn配置
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_CLASSPATH=`hadoop classpath`
注：如果hadoop是集成在CDH中，配置HADOOP_CONF_DIR=/etc/hadoop/conf
source /etc/profile #使环境变量生效
3、向yarn集群申请资源并提交任务
3.1 申请资源
(1)申请资源
bin/yarn-session.sh -nm wordCount -n 2
(2)查看申请的资源
使用命令查看
yarn application --list
yarn集群ui界面查看
yarn.webapp.ui2界面查看
3.2 提交wordCount计算作业
(1)提交作业到application id为application_1589204773282_0003的yarn session中
cd /usr/local/softwareinstall/flink-1.10.0
./bin/flink run -yid application_1589204773282_0003 ./examples/batch/WordCount.jar
(2)flink job ui界面观察作业执行信息
参考：
(1)YARN Setup
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/yarn_setup.html
(2)JobManager High Availability (HA)
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/jobmanager_high_availability.html
