CDH5.16安装flink1.10.0
1.采用parcels包的方式来安装flink
编译parcels请参考:制作Flink的Parcel包和csd文件
2.将parcels包拷贝到对应目录
lintong@master:/opt/cloudera/parcel-repo$ ls | grep FLINK FLINK-1.10.0-BIN-SCALA_2.11-xenial.parcel FLINK-1.10.0-BIN-SCALA_2.11-xenial.parcel.sha
csd包
lintong@master:/opt/cloudera/csd$ ls FLINK_ON_YARN-1.10.0.jar
3.安装
配置主机
配置flink集群参数
部署成功
flink集群部署后会在YARN上启动一个以flink用户运行的flink session,注意此时在此flink session中是不能认证成其他用户的,只能是flink用户,可以参考:Flink on Yarn with Kerberos
比如使用flink-scala-shell就会运行在此flink session中,而在里面认证是不会生效
部署成功
参考:Cloudera Manager中安装部署Flink服务
4.遇到的一些问题:
其中kerberos相关的参数如果集群没有启用kerberos的话可以不填写
如果有开启kerberos的话,需要在首先在os上添加flink用户
sudo groupadd flink sudo useradd flink -g flink -r --no-log-init -d /var/lib/flink sudo mkdir /var/lib/flink sudo chown flink:flink /var/lib/flink
然后创建生成flink用户的keytab
如果集成了ldap,需要在ldap中创建flink Group和People
如果重启flink集群的时候遇到
security.UserGroupInformation: PriviledgedActionException as:flink (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
需要先使用flink用户进行kinit认证
sudo -u flink kinit -kt /etc/keytab/flink.keytab flink/master@HADOOP.COM
启动flink-scala-shell验证
lintong@master:/opt/cloudera/parcels/FLINK/lib/flink/bin$ ./start-scala-shell.sh yarn
如果遇到
Exception in thread "main" java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z
请下载 jline-2.14.3.jar包 到flink的lib目录
lintong@master:/opt/cloudera/parcels/FLINK/lib/flink/lib$ wget https://repo1.maven.org/maven2/jline/jline/2.14.3/jline-2.14.3.jar
如果遇到
Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exceptions.YarnException at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 7 more
请下载 flink-shaded-hadoop-2-uber-2.7.5-7.0.jar包,放到flink的lib目录下
lintong@master:/opt/cloudera/parcels/FLINK/lib/flink/lib$ wget https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.7.5-7.0/flink-shaded-hadoop-2-uber-2.7.5-7.0.jar
读取文件并打印
scala> val dataSet = benv.readTextFile("hdfs:///user/hive/warehouse/test/000000_0") scala> dataSet.print()
本文只发表于博客园和tonglin0325的博客,作者:tonglin0325,转载请注明原文链接:https://www.cnblogs.com/tonglin0325/p/4576965.html