Hadoop URL读取数据
声明:代码来自《Hadoop应用技术开发详解》4.7.1,版权归作者所有。
1. 原理
通过URL的setURLStreamHandlerFactory方法设置Hadoop文件系统的URLStreamHandlerFactory的实现类FsUrlStreamHandlerFactory,这个方法在一个java虚拟机中只能被调用一次,因此一般都是放在一个static块中执行。
2. 代码
file: hdfs\HDFSURLReader.java
import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
import org.apache.hadoop.io.IOUtils;
import java.io.InputStream;
import java.net.URL;
public class HDFSURLReader {
static {
URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
}
public static void main(String[] args) {
InputStream inputStream = null;
try {
inputStream = new URL(args[0]).openStream();
IOUtils.copyBytes(inputStream, System.out, 1024, false);
} catch (Exception e) {
IOUtils.closeStream(inputStream);
}
}
}
3. 运行结果
[hadoop@master hdfs]$ hdfs dfs -cat hdfs://master:9000/user/hadoop/study/mr/WordCount/input/file1.txt
Hello, i love coding
are you ok?
Hello, i love hadoop
areyou ok?
[hadoop@master hdfs]$ hadoop jar HDFSURLReader.jar hdfs://master:9000/user/hadoop/study/mr/WordCount/input/file1.txt
Hello, i love coding
are you ok?
Hello, i love hadoop
areyou ok?