Hadoop本地编写的jar包放到集群执行时报错处理
错误描述:
020-03-24 22:45:23,204 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1585061045134_0001_01_000001 is : 1 2020-03-24 22:45:23,205 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1585061045134_0001_01_000001 and exit code: 1 ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at org.apache.hadoop.util.Shell.run(Shell.java:901) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:294) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:491) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:101) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
错误原因:查看日志发现有这样一条信息:conf.Configuration: resource-types.xml not found 还有这个: bad substitution 查了半天发现是提交任务出错了!
解决办法:
hadoop3的配置和hadoop2的配置基本相同,但是,在hadoop3中还需要在mapred-site.xml上新增一些东西:没有添加以下内容会找不到各种各样的类!添加之后问题解决。
<property> <name>yarn.app.mapreduce.am.env</name> <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value> </property> <property> <name>mapreduce.map.env</name> <value>HADOOP_MAPRED_HOME=${HADOP_HOME}</value> </property> <property> <name>mapreduce.reduce.env</name> <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value> </property> <property> <name>mapreduce.application.classpath</name> <value> ${HADOOP_HOME}/etc/hadoop, ${HADOOP_HOME}/share/hadoop/common/*, ${HADOOP_HOME}/share/hadoop/common/lib/*, ${HADOOP_HOME}/share/hadoop/hdfs/*, ${HADOOP_HOME}/share/hadoop/hdfs/lib/*, ${HADOOP_HOME}/share/hadoop/mapreduce/*, ${HADOOP_HOME}/share/hadoop/mapreduce/lib/*, ${HADOOP_HOME}/share/hadoop/yarn/*, ${HADOOP_HOME}/share/hadoop/yarn/lib/* </value>
</property>
唯有热爱方能抵御岁月漫长。