BigData<5>_HDFS API编程
一、开发环境搭建
-
选择IDEA,JAVA,使用Maven来管理项目
-
修改om.xml配置文件,加入Hadoop依赖包
<!--引入cdh的仓库-->
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
</repository>
</repositories>
<properties>
<!--定义Hadoop版本-->
<hadoop.version>2.6.0-cdh5.15.1</hadoop.version>
</properties>
<dependencies>
<!--添加Hadoop的依赖包-->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<!--添加junit的依赖包-->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>test</scope>
</dependency>
</dependencies>
二、使用Java API 操作HDFS文件系统
- 关键点:
- new 一个Configuration
- 获取FileSystem
- HDFS API操作
Configuration configuration = new Configuration();
FileSystem fileSystem = FileSystem.get(new URI("hdfs://hadoop000:8020"), configuration, "hadoop000");
Path path = new Path("/hdfsapi/test");
fileSystem.mkdirs(path);
- jUnit 封装