Cannot initialize Cluster. Please check your configuration for mapreduce.framework .name and the cor

背景

利用ambari搭建的新环境，跑数据出现了不少问题，但如下问题困扰了很长时间，直到今天才得以解决，每次报错。按照网上的各种方式都不行。我知道问题点肯定在spark2.3.1 集成hive3.1.0的版本问题上，因为hive3.1.0新增了很多功能，如事务等，发布时间没有长时间的积累，出问题很容易不受控制。

环境

采用ambari2.7.1 + spark2.3.1 + hadoop3.1.1 + hive3.1.0

scala2.11.8, jdk1.8

代码

// 可以正常打印

df.show(10, truncate = false)

df.createOrReplaceTempView("tmp_logistics_track")

// 可以正常打印

spark.sql("select * from tmp_logistics_track").show(20, truncate = false)

// 以下是报错所在行

spark.sql(

s"""insert overwrite table ods_common.ods_common_logistics_track_d PARTITION(p_day='$day')

|select dc_id, source_order_id, order_id, tracking_number, warehouse_code, user_id, channel_name,

|creation_time, update_time, sync_time, last_tracking_time, tracking_change_time, next_tracking_time,

|tms_nti_time, tms_oc_time, tms_as_time, tms_pu_time, tms_it_time, tms_od_time, tms_wpu_time, tms_rt_time,

|tms_excp_time, tms_fd_time, tms_df_time, tms_un_time, tms_np_time from tmp_logistics_track

|""".stripMargin)

部署脚本

#!/usr/bin/env bash

################# LOAD DATA TO HIVE ######################

#windows 编辑shell 需要修改编码为Unix

#命令

#set ff=unix

#SPARK_JARS_BASE_PATH=/home/isuhadoop/ark_data_bin/tag_batch/KafkaToHive/external_jar

set -x

v_proc_date=$1

#v_proc_date=$(date -d '-0 day' '+%Y%m%d')

echo "-----1: $1"

echo "-----2: $2"

# shell的使用的磁盘根目录

SHELL_ROOT_DIR=/home/ztsauser/limin_work/warehouse

v_exec_time=`date "+%Y%m%d%H"`

##日志目录

v_log_dir=${SHELL_ROOT_DIR}/logs/LogisticsTrackSourceProcess_${v_proc_date}.log

#如果没传参数，退出程序

if [[ "$v_proc_date" = "" ]]

then

echo "没有传入参数，即将退出程序》》》》》》" > ${v_log_dir}

exit

echo "调用脚本开始》》》》》》" > ${v_log_dir}

export HADOOP_USER_NAME=hive

/usr/hdp/current/spark2-client/bin/spark-submit --class zt.dc.bigdata.bp.process.warehouse.LogisticsTrackSourceProcess \

--name LogisticsTrackSourceProcess_${v_proc_date} \

--master yarn-cluster \

--queue default \

--deploy-mode cluster \

--num-executors 5 \

--executor-cores 2 \

--executor-memory 18g \

--files ${SHELL_ROOT_DIR}/config/hive-site.xml \

--jars ${SHELL_ROOT_DIR}/jar/hadoop-distcp-3.1.1.3.0.1.0-187.jar \

${SHELL_ROOT_DIR}/jar/dc-bp-1.0-SNAPSHOT-shaded.jar ${v_proc_date} > ${v_log_dir} 2>&1

异常

Caused by: java.io.IOException: Cannot execute DistCp process: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework

.name and the correspond server addresses.

详细信息如下：

21/01/09 23:06:08 INFO Client: Application report for application_1610095260612_0149 (state: RUNNING)

21/01/09 23:06:09 INFO Client: Application report for application_1610095260612_0149 (state: RUNNING)

21/01/09 23:06:10 INFO Client: Application report for application_1610095260612_0149 (state: RUNNING)

21/01/09 23:06:11 INFO Client: Application report for application_1610095260612_0149 (state: RUNNING)

21/01/09 23:06:12 INFO Client: Application report for application_1610095260612_0149 (state: RUNNING)

21/01/09 23:06:13 INFO Client: Application report for application_1610095260612_0149 (state: RUNNING)

21/01/09 23:06:14 INFO Client: Application report for application_1610095260612_0149 (state: RUNNING)

21/01/09 23:06:15 INFO Client: Application report for application_1610095260612_0149 (state: RUNNING)

21/01/09 23:06:16 INFO Client: Application report for application_1610095260612_0149 (state: RUNNING)

21/01/09 23:06:17 INFO Client: Application report for application_1610095260612_0149 (state: RUNNING)

21/01/09 23:06:18 INFO Client: Application report for application_1610095260612_0149 (state: RUNNING)

21/01/09 23:06:19 INFO Client: Application report for application_1610095260612_0149 (state: FINISHED)

21/01/09 23:06:19 INFO Client:

client token: N/A

diagnostics: User class threw exception: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source hdfs

://ztcluster/warehouse/tablespace/managed/hive/ods_common.db/ods_common_logistics_track_d/.hive-staging_hive_2021-01-09_23-05-39_625_275694820341612468-1/-ext-10000 t

o destination hdfs://ztcluster/warehouse/tablespace/managed/hive/ods_common.db/ods_common_logistics_track_d/p_day=20190321;

at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)

at org.apache.spark.sql.hive.HiveExternalCatalog.loadPartition(HiveExternalCatalog.scala:843)

at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:249)

at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:99)

at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)

at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)

at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:115)

at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)

at org.apache.spark.sql.Dataset$$anonfun$6.apply(Datas

destination hdfs://ztcluster/warehouse/tablespace/managed/hive/ods_common.db/ods_common_logistics_track_d/p_day=20190321;
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
at org.apache.spark.sql.hive.HiveExternalCatalog.loadPartition(HiveExternalCatalog.scala:843)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:249)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:99)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:115)
at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3259)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3258)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:190)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:75)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
at zt.dc.bigdata.bp.process.warehouse.LogisticsTrackSourceProcess$.main(LogisticsTrackSourceProcess.scala:122)
at zt.dc.bigdata.bp.process.warehouse.LogisticsTrackSourceProcess.main(LogisticsTrackSourceProcess.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source hdfs://ztcluster/warehouse/tablespace/managed/hive/ods_common.db/ods_common_logisti
cs_track_d/.hive-staging_hive_2021-01-09_23-05-39_625_275694820341612468-1/-ext-10000 to destination hdfs://ztcluster/warehouse/tablespace/managed/hive/ods_common.db/
ods_common_logistics_track_d/p_day=20190321
at org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:4057)
at org.apache.hadoop.hive.ql.metadata.Hive.getHiveException(Hive.java:4012)
at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:4007)
at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:4372)
at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1962)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.sql.hive.client.Shim_v3_0.loadPartition(HiveShim.scala:1275)
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadPartition$1.apply$mcV$sp(HiveClientImpl.scala:747)
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadPartition$1.apply(HiveClientImpl.scala:745)
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$loadPartition$1.apply(HiveClientImpl.scala:745)
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:278)
at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:216)
at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:215)
at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:261)
at org.apache.spark.sql.hive.client.HiveClientImpl.loadPartition(HiveClientImpl.scala:745)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadPartition$1.apply$mcV$sp(HiveExternalCatalog.scala:855)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadPartition$1.apply(HiveExternalCatalog.scala:843)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$loadPartition$1.apply(HiveExternalCatalog.scala:843)
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
... 21 more
Caused by: java.io.IOException: Cannot execute DistCp process: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework
.name and the correspond server addresses.
at org.apache.hadoop.hive.shims.Hadoop23Shims.runDistCp(Hadoop23Shims.java:1151)
at org.apache.hadoop.hive.common.FileUtils.distCp(FileUtils.java:643)
at org.apache.hadoop.hive.common.FileUtils.copy(FileUtils.java:625)
at org.apache.hadoop.hive.common.FileUtils.copy(FileUtils.java:600)
at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:3921)
... 40 more
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:116)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:109)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:102)
at org.apache.hadoop.tools.DistCp.createMetaFolderPath(DistCp.java:410)
at org.apache.hadoop.tools.DistCp.<init>(DistCp.java:116)
at org.apache.hadoop.hive.shims.Hadoop23Shims.runDistCp(Hadoop23Shims.java:1141)
... 44 more
ApplicationMaster host: 192.168.81.58
ApplicationMaster RPC port: 0
queue: default
start time: 1610204535333
final status: FAILED
tracking URL: http://szch-ztn-dc-bp-pro-192-168-81-57:8088/proxy/application_1610095260612_0149/
user: hive
Exception in thread "main" org.apache.spark.SparkException: Application application_1610095260612_0149 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1269)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1627)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

解决方案

根据网上的方案我进行了以下尝试

尝试一(未解决)

尝试了在客户添加如下pom依赖，并未解决

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-jobclient</artifactId>
<version>${hadoop.version}</version>
<scope>${scopetype}</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
<scope>${scopetype}</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${hadoop.version}</version>
<scope>${scopetype}</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
<scope>${scopetype}</scope>
</dependency>还有mapreduce-jobclient,mapreduce等，注意scope一定要改成compile不能是test，不然还是会报错

尝试二（未解决）

修改hdfs-site.xml中

fs.hdfs.impl.disable.cache属性为true

尝试三（未解决）

关闭hive的事务功能

关闭hdp 3.0 创建表自动为acid表的参数：

hive.create.as.insert.only=false

metastore.create.as.acid=false

hive.strict.managed.tables=false

hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager

hive.support.concurrency=false

// hive.stats.autogather属性要设置成true, 否则在执行sql时，会报异常

hive.stats.autogather=true

尝试四（未解决）

检查mapred-site.xml文件的mapreduce.framework.name属性是否为yarn

如果不是改成local再尝试

尝试五（解决）

在找遍了网上所有的方法无解后，想到了一个点，之前在使用hive3.0时，如果分区事先创建好了，通过第三方api或spark客户端写数据时有问题，

所以便尝试了，让创建分区的方法显示声明在代码中，再次尝试，问题解决。但为什么会这样，到目前还未知。

附代码：

// 删除分区
spark.sql(s"alter table ods_common.ods_common_logistics_track_d drop if exists partition(p_day='$day')")
// 添加分区
spark.sql(s"alter table ods_common.ods_common_logistics_track_d add if not exists partition (p_day='$day')")
df.show(10, truncate = false)
df.createOrReplaceTempView("tmp_logistics_track")
spark.sql("select * from tmp_logistics_track").show(20, truncate = false)
spark.sql(
s"""insert overwrite table ods_common.ods_common_logistics_track_d PARTITION(p_day='$day')
|select dc_id, source_order_id, order_id, tracking_number, warehouse_code, user_id, channel_name,
|creation_time, update_time, sync_time, last_tracking_time, tracking_change_time, next_tracking_time,
|tms_nti_time, tms_oc_time, tms_as_time, tms_pu_time, tms_it_time, tms_od_time, tms_wpu_time, tms_rt_time,
|tms_excp_time, tms_fd_time, tms_df_time, tms_un_time, tms_np_time from tmp_logistics_track
|""".stripMargin)

本文转自：https://blog.csdn.net/arlanhon/article/details/112480999?spm=1001.2101.3001.6650.6&utm_medium=distribute.pc_relevant.none-task-blog-2%7Edefault%7EESLANDING%7Edefault-6-112480999-blog-98077988.pc_relevant_landingrelevant&depth_1-utm_source=distribute.pc_relevant.none-task-blog-2%7Edefault%7EESLANDING%7Edefault-6-112480999-blog-98077988.pc_relevant_landingrelevant&utm_relevant_index=7