[b0009] 玩Hadoop中碰到的各种错误

1. Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found

hadoop@ssmaster:~/java_program$ hadoop  jar hadoop_mapr_wordcount.jar  /input /output 
16/10/24 08:22:41 INFO client.RMProxy: Connecting to ResourceManager at ssmaster/192.168.249.144:8032
16/10/24 08:22:42 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/10/24 08:22:42 WARN mapreduce.JobResourceUploader: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
16/10/24 08:22:42 INFO input.FileInputFormat: Total input paths to process : 1
16/10/24 08:22:43 INFO mapreduce.JobSubmitter: number of splits:1
16/10/24 08:22:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1477315002921_0003
16/10/24 08:22:44 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
16/10/24 08:22:44 INFO impl.YarnClientImpl: Submitted application application_1477315002921_0003
16/10/24 08:22:44 INFO mapreduce.Job: The url to track the job: http://ssmaster:8088/proxy/application_1477315002921_0003/
16/10/24 08:22:44 INFO mapreduce.Job: Running job: job_1477315002921_0003
16/10/24 08:22:57 INFO mapreduce.Job: Job job_1477315002921_0003 running in uber mode : false
16/10/24 08:22:57 INFO mapreduce.Job:  map 0% reduce 0%
16/10/24 08:23:04 INFO mapreduce.Job: Task Id : attempt_1477315002921_0003_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2154)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:742)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2060)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2152)
    ... 8 more

16/10/24 08:23:12 INFO mapreduce.Job: Task Id : attempt_1477315002921_0003_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2154)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:742)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2060)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2152)
    ... 8 more

16/10/24 08:23:21 INFO mapreduce.Job: Task Id : attempt_1477315002921_0003_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2154)
    at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:186)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:742)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class mp.filetest.WordCount2$Map not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2060)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2152)
View Code
背景：Hadoop2.6.4 Ubuntu， hadoop jar xxx.jar  运行程序时报错

原因：在代码中没有指定执行类

解决：

添加 job.setJarByClass(WordCount2.class);
解决

 

2  Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://ssmaster:9000/data/paper.txt, expected: file:///

Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://ssmaster:9000/data/paper.txt, expected: file:///
    at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:645)
    at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80)
    at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:532)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:750)
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:527)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:140)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
    at hadoop.hdfs.Hdfs_Download.main(Hdfs_Download.java:36)
View Code
背景：Hadoop2.6.4 Ubuntu, windows 7 eclipse 已经配置好win下运行hadoop插件环境，参考[0008] ，运行hdfs时报错 (下载一个文件到windows本地)

       路径设置如下 src = new Path("hdfs://ssmaster:9000/data/paper.txt" );

       out = new FileOutputStream ("c:\\paper.txt");

解决：

     将 core-site.xml，hdfs-site.xml，mapred-site.xml 复制到eclipse工程的src目录下
解决

 

3  log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
View Code
背景： Hadoop2.6.4 Ubuntu, windows 7 eclipse 运行程序

解决： 工程下的src目录新建一个文件 log4j.properties，添加内容（网上找)
解决

4 main"java.lang.UnsatisfiedLinkError:org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

 

背景：window 7 64，eclipse 执行 mapreduce wordcount 程序

原因：win 下开发的问题

解决：

源码包 hadoop-2.6.4-src.tar.gz解压，hadoop-2.6.4-src\hadoop-common-project\hadoop-common\src\main\java\org\apache\hadoop\io\nativeio下NativeIO.java 复制到对应的Eclipse的project

       修改如下地方

 

如果没有解决，按照 Y.2.1 中配置

[0008]  http://www.cnblogs.com/sunzebo/p/5991563.html

 

posted @ 2016-10-24 23:33  sunzebo  阅读(3932)  评论(0)  编辑  收藏  举报