Spark以yarn方式运行时抛出异常
Spark以yarn方式运行时抛出异常:
cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!
解决方案:
主要是给节点分配的内存少,yarn kill了spark application。
1.停止hadoop集群
[root@node06 bin]# stop-all.sh
2.给yarn-site.xml增加配置:
<property> <name>yarn.nodemanager.pmem-check-enabled</name> <value>false</value> </property> <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> <description>Whether virtual memory limits will be enforced for containers</description> </property> <property> <name>yarn.nodemanager.vmem-pmem-ratio</name> <value>4</value> <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description> </property>
3.将master上将l修改后的yarn-site.xm文件覆盖到各个slaves节点
[root@node06 hadoop]# scp yarn-site.xml node07:`pwd` yarn-site.xml 100% 1565 1.5KB/s 00:00 [root@node06 hadoop]# scp yarn-site.xml node08:`pwd` yarn-site.xml 100% 1565 1.5KB/s 00:00 [root@node06 hadoop]# scp yarn-site.xml node09:`pwd`
4.重新启动hadoop集群
[root@node06 hadoop]# start-all.sh
5.重新向yarn提交spark任务
[root@node06 bin]# ./spark-submit --master yarn --class org.apache.spark.examples.SparkPi ../lib/spark-examples-1.6.0-hadoop2.6.0.jar 100