【Flink系列四】构建实时计算平台——Flink SQLClient启动失败的问题笔记

合集 - 助力长期平稳运行大数据作业(23)

1.【Flink系列二十】Flink Example AsyncIOExample long running 应用程序的应用2023-10-262.【Flink系列二十一】深入理解 JVM的类型加载约束，解决 Flink 类型加载冲突问题的通用方法2023-12-293.【Flink系列十九】Flink 作业Hadoop 依赖冲突解决NoSuchMethodError2023-09-184.HDFS_DELEGATION_TOKEN过期的问题解决汇总2023-09-125.【Flink系列十八】History Server 重新登场，如何实现Yarn日志集成2023-05-316.【Flink系列十七】Flink 最新yarn-application和yarn-per-job部署模式的Classpath传递分析2023-05-297.【Flink系列十六】PrometheusPushGatewayReporter 限流压力过大解决2022-08-038.【Flink系列十五】FLINK-24950 registerTemporaryFunction NullPointerException2022-04-229.【Flink系列十四】Flink JVM参数不生效的问题分析和解决2022-04-1110.【Flink系列十三】Flink-Kafka-Connector KafkaSource FlinkKafkaConsumer没有上报指标2022-03-2411.【Flink系列十二】使用OpenResty 在InfluxDB协议层拦截Flink指标2022-03-1312.【Flink系列十一】FlinkSQL Gateway以及支持Kerberos多租户的实现思路2022-01-2613.【Flink系列十】Flink作业提交过程的调试和诊断2021-11-0214.【Flink系列九】Flink 作业提交遇到的问题记录以及原理2021-11-0115.【Flink系列八】构建实时计算平台——动态加载Catalog中的UDF2021-05-1016.【Flink系列零】构建实时计算平台——FlinkSQL 作业菜鸟笔记2021-05-1017.【Flink系列七】构建实时计算平台——校验FlinkSQL2021-04-1918.【Flink系列六】构建实时计算平台——Flink 1.10+通过Kerberos连接HiveCatalog2021-03-1519.【Flink系列五】构建实时计算平台——flink-connector-hive连接HiveMetastore遇到问题2021-01-08

20.【Flink系列四】构建实时计算平台——Flink SQLClient启动失败的问题笔记2021-01-05

21.【Flink系列三】构建实时计算平台——特别篇，Influxdb Java客户端访问指标数据2020-10-2922.【Flink系列二】构建实时计算平台——特别篇，用InfluxDb收集Flink Metrics2020-10-2323.【Flink系列一】构建实时计算平台——Flink开启和恢复Checkpoint2020-10-23

收起

问题

报错
org.apache.flink.table.api.NoMatchingTableFactoryException: Could not find a suitable table factory for 'org.apache.flink.table.factories.CatalogFactory' in the classpath.

分析

1. 此台服务器没有完整的HADOOP_CONF_DIR,HADOOP_CLASSPATH等环境变量。
2. flink-conf.yml没有经过任何修改
3. FLINK_DIR/lib下没有添加任何jar。
4. 在conf/sql-client-defaults.yml中配置了Catalog.

# Define catalogs here.

catalogs:  # empty list
 - name: pfc
   type: hive
   hive-conf-dir: /etc/hive/conf/
   default-database: flink

解决问题

查阅 Flink-HIVE 文档

老实将图中的jar 添加到 FLINK_DIR/lib下面。

否则出现：

Exception in thread "main" org.apache.flink.table.client.SqlClientException: Unexpected exception. This is a bug. Please consider filing an issue.
	at org.apache.flink.table.client.SqlClient.main(SqlClient.java:208)
Caused by: org.apache.flink.table.client.gateway.SqlExecutionException: Could not create execution context.
	at org.apache.flink.table.client.gateway.local.ExecutionContext$Builder.build(ExecutionContext.java:878)
	at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:226)
	at org.apache.flink.table.client.SqlClient.start(SqlClient.java:108)
	at org.apache.flink.table.client.SqlClient.main(SqlClient.java:196)
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/AlreadyExistsException
	at org.apache.flink.table.catalog.hive.factories.HiveCatalogFactory.createCatalog(HiveCatalogFactory.java:89)
	at org.apache.flink.table.client.gateway.local.ExecutionContext.createCatalog(ExecutionContext.java:384)
	at org.apache.flink.table.client.gateway.local.ExecutionContext.lambda$null$5(ExecutionContext.java:634)
	at java.util.HashMap.forEach(HashMap.java:1288)
	at org.apache.flink.table.client.gateway.local.ExecutionContext.lambda$initializeCatalogs$6(ExecutionContext.java:633)
	at org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:266)
	at org.apache.flink.table.client.gateway.local.ExecutionContext.initializeCatalogs(ExecutionContext.java:632)
	at org.apache.flink.table.client.gateway.local.ExecutionContext.initializeTableEnvironment(ExecutionContext.java:529)
	at org.apache.flink.table.client.gateway.local.ExecutionContext.<init>(ExecutionContext.java:185)
	at org.apache.flink.table.client.gateway.local.ExecutionContext.<init>(ExecutionContext.java:138)
	at org.apache.flink.table.client.gateway.local.ExecutionContext$Builder.build(ExecutionContext.java:867)
	... 3 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.AlreadyExistsException
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 14 more

或者出现：

Exception in thread "main" org.apache.flink.table.client.SqlClientException: Unexpected exception. This is a bug. Please consider filing an issue.
	at org.apache.flink.table.client.SqlClient.main(SqlClient.java:208)
Caused by: org.apache.flink.table.client.gateway.SqlExecutionException: Could not create execution context.
	at org.apache.flink.table.client.gateway.local.ExecutionContext$Builder.build(ExecutionContext.java:878)
	at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:226)
	at org.apache.flink.table.client.SqlClient.start(SqlClient.java:108)
	at org.apache.flink.table.client.SqlClient.main(SqlClient.java:196)
Caused by: java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.apache.flink.table.catalog.hive.client.HiveShimV100.getHiveMetastoreClient(HiveShimV100.java:97)
	at org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.createMetastoreClient(HiveMetastoreClientWrapper.java:245)
	at org.apache.flink.table.catalog.hive.client.HiveMetastoreClientWrapper.<init>(HiveMetastoreClientWrapper.java:76)
	at org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory.create(HiveMetastoreClientFactory.java:35)
	at org.apache.flink.table.catalog.hive.HiveCatalog.open(HiveCatalog.java:245)
	at org.apache.flink.table.catalog.CatalogManager.registerCatalog(CatalogManager.java:190)
	at org.apache.flink.table.api.internal.TableEnvironmentImpl.registerCatalog(TableEnvironmentImpl.java:338)
	at org.apache.flink.table.client.gateway.local.ExecutionContext.lambda$null$5(ExecutionContext.java:635)
	at java.util.HashMap.forEach(HashMap.java:1288)
	at org.apache.flink.table.client.gateway.local.ExecutionContext.lambda$initializeCatalogs$6(ExecutionContext.java:633)
	at org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:266)
	at org.apache.flink.table.client.gateway.local.ExecutionContext.initializeCatalogs(ExecutionContext.java:632)
	at org.apache.flink.table.client.gateway.local.ExecutionContext.initializeTableEnvironment(ExecutionContext.java:529)
	at org.apache.flink.table.client.gateway.local.ExecutionContext.<init>(ExecutionContext.java:185)
	at org.apache.flink.table.client.gateway.local.ExecutionContext.<init>(ExecutionContext.java:138)
	at org.apache.flink.table.client.gateway.local.ExecutionContext$Builder.build(ExecutionContext.java:867)
	... 3 more
Caused by: java.lang.ClassNotFoundException: com.facebook.fb303.FacebookService$Iface
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 31 more

另外

Flink 连接Hive的时候，需要通过HDFS配置文件来知道Hive服务的位置，所以还需要Hadoop的依赖。

通过查看 FLINK_DIR/bin/config.sh发现，HADOOP_CLASSPATH非常重要，仅 HADOOP_CONF_DIR似乎不够。

因此这样执行便可以启动 Flink SQL CLient:

FLINK_DIR/bin $>HADOOP_CLASSPATH=`hadoop classpath` ./sql-client.sh embedded

于是这样就启动成功了