Hadoop3.1.1运行自带例子wordcount发生的错误
错误 1
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster Please check whether your etc/hadoop/mapred-site.xml contains the below configuration: <property> <name>yarn.app.mapreduce.am.env</name> <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value> </property> <property> <name>mapreduce.map.env</name> <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value> </property> <property> <name>mapreduce.reduce.env</name> <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value> </property>
解决方法
在配置文件中添加hadoop的根目录即可，如：
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>yarn.app.mapreduce.am.env</name> <value>HADOOP_MAPRED_HOME=/usr/local/hadoop/hadoop-3.1.1</value> </property> <property> <name>mapreduce.map.env</name> <value>HADOOP_MAPRED_HOME=/usr/local/hadoop/hadoop-3.1.1</value> </property> <property> <name>mapreduce.reduce.env</name> <value>HADOOP_MAPRED_HOME=/usr/local/hadoop/hadoop-3.1.1</value> </property> </configuration>
错误 2
2018-12-09 22:01:31,884 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032 2018-12-09 22:01:32,752 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1544410258618_0005 2018-12-09 22:01:33,056 INFO input.FileInputFormat: Total input files to process : 1 2018-12-09 22:01:33,621 INFO mapreduce.JobSubmitter: number of splits:1 2018-12-09 22:01:33,664 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled 2018-12-09 22:01:34,306 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1544410258618_0005 2018-12-09 22:01:34,307 INFO mapreduce.JobSubmitter: Executing with tokens: [] 2018-12-09 22:01:34,595 INFO conf.Configuration: resource-types.xml not found 2018-12-09 22:01:34,595 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 2018-12-09 22:01:34,715 INFO impl.YarnClientImpl: Submitted application application_1544410258618_0005 2018-12-09 22:01:34,772 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1544410258618_0005/ 2018-12-09 22:01:34,773 INFO mapreduce.Job: Running job: job_1544410258618_0005 2018-12-09 22:01:43,122 INFO mapreduce.Job: Job job_1544410258618_0005 running in uber mode : false 2018-12-09 22:01:43,123 INFO mapreduce.Job: map 0% reduce 0% 2018-12-09 22:01:57,528 INFO mapreduce.Job: Task Id : attempt_1544410258618_0005_m_000000_0, Status : FAILED [2018-12-09 22:01:54.252]Container [pid=10353,containerID=container_1544410258618_0005_01_000002] is running 523131392B beyond the 'VIRTUAL' memory limit. Current usage: 257.3 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container. Dump of the process-tree for container_1544410258618_0005_01_000002 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 10353 10352 10353 10353 (bash) 0 0 115896320 173 /bin/bash -c /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_0 2 1>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000002/stdout 2>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000002/stderr |- 10364 10353 10353 10353 (java) 624 174 2662092800 65693 /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_0 2 [2018-12-09 22:01:55.961]Container killed on request. Exit code is 143 [2018-12-09 22:01:55.961]Container exited with a non-zero exit code 143. 2018-12-09 22:02:07,613 INFO mapreduce.Job: Task Id : attempt_1544410258618_0005_m_000000_1, Status : FAILED [2018-12-09 22:02:06.339]Container [pid=10421,containerID=container_1544410258618_0005_01_000003] is running 521972224B beyond the 'VIRTUAL' memory limit. Current usage: 136.2 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container. Dump of the process-tree for container_1544410258618_0005_01_000003 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 10430 10421 10421 10421 (java) 346 100 2660933632 34552 /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_1 3 |- 10421 10419 10421 10421 (bash) 0 0 115896320 306 /bin/bash -c /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_1 3 1>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000003/stdout 2>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000003/stderr [2018-12-09 22:02:06.409]Container killed on request. Exit code is 143 [2018-12-09 22:02:06.420]Container exited with a non-zero exit code 143. 2018-12-09 22:02:19,533 INFO mapreduce.Job: map 100% reduce 0% 2018-12-09 22:02:19,534 INFO mapreduce.Job: Task Id : attempt_1544410258618_0005_m_000000_2, Status : FAILED [2018-12-09 22:02:16.854]Container [pid=10477,containerID=container_1544410258618_0005_01_000004] is running 551004672B beyond the 'VIRTUAL' memory limit. Current usage: 272.7 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container. Dump of the process-tree for container_1544410258618_0005_01_000004 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 10486 10477 10477 10477 (java) 412 63 2689966080 69764 /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_2 4 |- 10477 10475 10477 10477 (bash) 0 0 115896320 50 /bin/bash -c /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_2 4 1>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000004/stdout 2>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000004/stderr [2018-12-09 22:02:18.409]Container killed on request. Exit code is 143 [2018-12-09 22:02:18.410]Container exited with a non-zero exit code 143. 2018-12-09 22:02:20,541 INFO mapreduce.Job: map 0% reduce 0% 2018-12-09 22:02:33,619 INFO mapreduce.Job: map 100% reduce 100% 2018-12-09 22:02:35,639 INFO mapreduce.Job: Job job_1544410258618_0005 failed with state FAILED due to: Task failed task_1544410258618_0005_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 2018-12-09 22:02:35,794 INFO mapreduce.Job: Counters: 13 Job Counters Failed map tasks=4 Killed reduce tasks=1 Launched map tasks=4 Other local map tasks=3 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=40350 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=40350 Total vcore-milliseconds taken by all map tasks=40350 Total megabyte-milliseconds taken by all map tasks=41318400 Map-Reduce Framework CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0
解决方法
添加内存
