随笔 - 597  文章 - 4  评论 - 445  阅读 - 424万

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.io.IOException: No FileSystem for sc F

1、执行脚本程序报如下所示的错误:

复制代码
 1 [hadoop@slaver1 script_hadoop]$  hadoop jar web_click_mr_hive.jar com.bie.hive.mr.ClickStreamThree /home/hadoop/data_hadoop/weblog/preprocess/output/2018-05-19 /home/hadoop/data_hadoop/weblog/preprocess/click_pv_out/2018-05-19
 2 18/05/20 17:09:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 3 Exception in thread "main" java.io.IOException: No FileSystem for scheme: C
 4     at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
 5     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
 6     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
 7     at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
 8     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
 9     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
10     at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
11     at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:498)
12     at com.bie.hive.mr.ClickStreamThree.main(ClickStreamThree.java:207)
13     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
14     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
15     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16     at java.lang.reflect.Method.invoke(Method.java:606)
17     at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
18 [hadoop@slaver1 script_hadoop]$  hadoop jar web_click_mr_hive.jar com.bie.hive.mr.ClickStreamThree hdfs://slaver1:9000/home/hadoop/data_hadoop/weblog/preprocess/output/2018-05-19 hdfs://slaver1:9000/home/hadoop/data_hadoop/weblog/preprocess/click_pv_out/2018-05-19
19 18/05/20 17:10:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20 Exception in thread "main" java.io.IOException: No FileSystem for scheme: C
21     at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
22     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
23     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
24     at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
25     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
26     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
27     at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
28     at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:498)
29     at com.bie.hive.mr.ClickStreamThree.main(ClickStreamThree.java:207)
30     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
31     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
32     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
33     at java.lang.reflect.Method.invoke(Method.java:606)
34     at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
35 [hadoop@slaver1 script_hadoop]$ 
复制代码

2、猛一看,WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable,百度一搜这个错误,啥编译32/64位的问题,也有说改配置文件的问题,也许有的可能是这样改就解决问题了,而我想说,出现问题,看看报错原因,也许就可以很快的解决问题了。

然后呢,看下代码,是如下所示导致的错误,将下面的注释了,代码换成上面的就可以使用脚本程序执行代码了:

FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

//FileInputFormat.setInputPaths(job, new Path("C:/weblog/output"));
//FileOutputFormat.setOutputPath(job, new Path("c:/weblog/pageviews"));

 

posted on   别先生  阅读(1362)  评论(0编辑  收藏  举报
编辑推荐:
· 如何编写易于单元测试的代码
· 10年+ .NET Coder 心语,封装的思维:从隐藏、稳定开始理解其本质意义
· .NET Core 中如何实现缓存的预热?
· 从 HTTP 原因短语缺失研究 HTTP/2 和 HTTP/3 的设计差异
· AI与.NET技术实操系列:向量存储与相似性搜索在 .NET 中的实现
阅读排行:
· 周边上新:园子的第一款马克杯温暖上架
· Open-Sora 2.0 重磅开源!
· 分享 3 个 .NET 开源的文件压缩处理库,助力快速实现文件压缩解压功能!
· Ollama——大语言模型本地部署的极速利器
· DeepSeek如何颠覆传统软件测试?测试工程师会被淘汰吗?
历史上的今天:
2017-05-20 作为程序猿/媛的你在5.20这一天都干了什么???
< 2025年3月 >
23 24 25 26 27 28 1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31 1 2 3 4 5

点击右上角即可分享
微信分享提示