spark-shell启动报错:Yarn application has already ended! It might have been killed or unable to launch application master

spark-shell不支持yarn cluster,以yarn client方式启动

spark-shell --master=yarn --deploy-mode=client

启动日志,错误信息如下

 

其中“Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME”,只是一个警告,官方的解释如下:

大概是说:如果 spark.yarn.jars 和 spark.yarn.archive都没配置,会把$SPAR_HOME/jars下面所有jar打包成zip文件,上传到每个工作分区,所以打包分发是自动完成的,没配置这俩参数没关系。

 

"Yarn application has already ended! It might have been killed or unable to launch application master",这个可是一个异常,打开mr管理页面,我的是 http://192.168.128.130/8088 ,

重点在红框处,2.2g的虚拟内存实际值,超过了2.1g的上限。也就是说虚拟内存超限,所以contrainer被干掉了,活都是在容器干的,容器被干掉了,还玩个屁。

解决方案

yarn-site.xml 增加配置:

2个配置2选一即可

 1 <!--以下为解决spark-shell 以yarn client模式运行报错问题而增加的配置,估计spark-summit也会有这个问题。2个配置只用配置一个即可解决问题,当然都配置也没问题-->
 2 <!--虚拟内存设置是否生效,若实际虚拟内存大于设置值 ,spark 以client模式运行可能会报错,"Yarn application has already ended! It might have been killed or unable to l"-->
 3 <property>
 4     <name>yarn.nodemanager.vmem-check-enabled</name>
 5     <value>false</value>
 6     <description>Whether virtual memory limits will be enforced for containers</description>
 7 </property>
 8 <!--配置虚拟内存/物理内存的值,默认为2.1,物理内存默认应该是1g,所以虚拟内存是2.1g-->
 9 <property>
10     <name>yarn.nodemanager.vmem-pmem-ratio</name>
11     <value>4</value>
12     <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
13 </property>
View Code

 

修改后,启动hadoop,spark-shell.

 

posted @ 2017-08-10 00:12  aj117  阅读(16253)  评论(0编辑  收藏  举报