Kylin 3.1.2 cube构建过程报错 org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
今天在使用kylin构建cube的时候报如下错误:
2022-10-09 16:45:00,932 ERROR [pool-12-thread-9] threadpool.DefaultScheduler:121 : ExecuteException job:f7b41a90-4c10-994d-e9b7-5c559e3d4a32
org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.exception.ExecuteException: java.lang.RuntimeException: java.io.IOException: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:201)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:119)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kylin.job.exception.ExecuteException: java.lang.RuntimeException: java.io.IOException: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:201)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:72)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180)
... 4 more
Caused by: java.lang.RuntimeException: java.io.IOException: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:83)
at org.apache.kylin.engine.mr.steps.CuboidJob.configureMapperInputFormat(CuboidJob.java:177)
at org.apache.kylin.engine.mr.steps.CuboidJob.run(CuboidJob.java:151)
at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:156)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180)
... 6 more
Caused by: java.io.IOException: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:97)
at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51)
at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:80)
... 10 more
Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:2079)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:2066)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1578)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1570)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208)
at com.sun.proxy.$Proxy73.getTable(Unknown Source)
at org.apache.hive.hcatalog.common.HCatUtil.getTable(HCatUtil.java:191)
at org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:105)
at org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:88)
at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95)
... 12 more
我在网上查找了相关的错误,发现找到的都是跟spark有关的,但是我的kylin没有使用spark进行构建。后来参考了网上的一些文章,能成功构建啦。
目前服务器的环境是:
hadoop 3.0.0-cdh6.3.2
hive 3.1.2
kylin 3.1.2
错误原因:应该是因为服务器的cdh最近升级到6.3.2,原先的hadoop2和hive2都升级为hadoop3和hive3,导致和kylin有点不兼容。
解决方法:
# 进入kylin的lib目录
cd $KYLIN_HOME/lib
# 将这两个jar包下载到该lib目录下
wget https://repo1.maven.org/maven2/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar
wget https://repo1.maven.org/maven2/commons-configuration/commons-configuration/1.3/commons-configuration-1.3.jar
# 创建目录并进入
mkdir -p $KYLIN_HOME/bin/hadoop3_jars/cdh6
cd $KYLIN_HOME/bin/hadoop3_jars/cdh6
# 将这两个jar包下载到该目录下
wget https://repo1.maven.org/maven2/org/codehaus/woodstox/stax2-api/3.1.4/stax2-api-3.1.4.jar
wget https://cwiki.apache.org/confluence/download/attachments/173081375/hive-exec-1.21.2.3.1.0.0-78.jar?api=v2
# 重启kylin
$KYLIN_HOME/bin/kylin.sh restart
以上步骤完成后,重新到web页面构建cube,发现能成功构建了。
作者:zoufh
出处:https://www.cnblogs.com/zoufh/p/17111325.html
版权:本作品采用「署名-非商业性使用-相同方式共享 4.0 国际」许可协议进行许可。
【推荐】国内首个AI IDE,深度理解中文开发场景,立即下载体验Trae
【推荐】编程新体验,更懂你的AI,立即体验豆包MarsCode编程助手
【推荐】抖音旗下AI助手豆包,你的智能百科全书,全免费不限次数
【推荐】轻量又高性能的 SSH 工具 IShell:AI 加持,快人一步
· 25岁的心里话
· 闲置电脑爆改个人服务器(超详细) #公网映射 #Vmware虚拟网络编辑器
· 基于 Docker 搭建 FRP 内网穿透开源项目(很简单哒)
· 零经验选手,Compose 一天开发一款小游戏!
· 通过 API 将Deepseek响应流式内容输出到前端