算法flink

和Yarn-Cluster模式不同，Session模式的资源在启动Yarn-Session时候就已经启动了，后续提交的作业全都在已申请的资源空间内运行，比较适合小而多的作业

# 启动yarn-session模式，不用启动flink集群

cm:

http://152.32.141.11:7180/cmf/login

 

登陆manager节点:

sudo su hdfs

yarn application -list

#查flink session的application id

 

 

#打开flink的web管理

http://manager.algorithm.opayride.com:8088/proxy/application_1572516566413_0919/#/job

#算法启动session:
/data/flink/bin/yarn-session.sh -n 20 -m 4 -qu flink -jm 1024 -tm 8192 -s 2 -d

-n 多少个容器
tm 每个容器的内存
-s 每个任务用多少slot

 

 

#在session上run job
cd /var/lib/hadoop-hdfs/src/oride-research/flink/flink-ufile
flink run -p 4 -c com.opay.research.hadoop.UfileJob target/flink-ufile-1.0.jar --prod

重启session:
1/ 在web管理界面上退出:
点running jobs,点运行的session job进去,把它cancel掉

 

2/ 用yarn kill application xxx把flink作业kill
3/ 重新启动seesion
/data/flink/bin/yarn-session.sh -n 20 -m 4 -qu flink -jm 1024 -tm 8192 -s 2 -d

 

参数:

 

bin/yarn-session.sh –help

Usage:
Required
-n,--container <arg> Number of YARN container to allocate (=Number of Task Managers)
Optional
-D <property=value> use value for given property
-d,--detached If present, runs the job in detached mode
-h,--help Help for the Yarn session CLI.
-id,--applicationId <arg> Attach to running YARN session
-j,--jar <arg> Path to Flink jar file
-jm,--jobManagerMemory <arg> Memory for JobManager Container with optional unit (default: MB)
-m,--jobmanager <arg> Address of the JobManager (master) to which to connect. Use this flag to connect to a different JobManager than the one specified in the configuration.
-n,--container <arg> Number of YARN container to allocate (=Number of Task Managers)
-nl,--nodeLabel <arg> Specify YARN node label for the YARN application
-nm,--name <arg> Set a custom name for the application on YARN
-q,--query Display available YARN resources (memory, cores)
-qu,--queue <arg> Specify YARN queue.
-s,--slots <arg> Number of slots per TaskManager
-sae,--shutdownOnAttachedExit If the job is submitted in attached mode, perform a best-effort cluster shutdown when the CLI is terminated abruptly, e.g., in response to a user interrupt, such
as typing Ctrl + C.
-st,--streaming Start Flink in streaming mode
-t,--ship <arg> Ship files in the specified directory (t for transfer)
-tm,--taskManagerMemory <arg> Memory per TaskManager Container with optional unit (default: MB)
-yd,--yarndetached If present, runs the job in detached mode (deprecated; use non-YARN specific option instead)
-z,--zookeeperNamespace <arg> Namespace to create the Zookeeper sub-paths for high availability mode

 

也可在启动时得到它的web地址

# 启动yarn-session模式，不用启动flink集群
[root@oldboy-node101 conf]# yarn-session.sh
2019-08-03 21:57:59,585 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Flink JobManager is now running on oldboy-node103:35244 with leader id 00000000-0000-0000-0000-000000000000.
JobManager Web Interface: http://oldboy-node103:35244