在IDEA中创建HDFS客户端并新建文件目录

 

前言

　　首先需要配置JAVA环境变量,搭建hadoop集群,实现HA,并启动集群环境

 

一 创建hdfs项目,导入pom文件

<dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>2.8.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.7.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.7.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.7.2</version>
        </dependency>
        <dependency>
            <groupId>com.sun</groupId>
            <artifactId>tools</artifactId>
            <version>1.8.0</version>
            <scope>system</scope>
            <systemPath>${env.JAVA_HOME}/lib/tools.jar</systemPath>
        </dependency>

    </dependencies>

注意:如果遇到导入jdk.tools依赖加载失败时,参考这里.

 

二 创建Client

package com.lxz.hdfs;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.junit.Test;


import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;

public class HdfsClient {

    @Test
    public void testMkdirs() throws IOException,InterruptedException, URISyntaxException{
        //1.获取文件系统
        Configuration configuration = new Configuration();
        //2.配置在集群上运行
        configuration.set("fs.defaultFS","hdfs://hadoop1:9000"); # hadoop需要读者写自己的
//        FileSystem fs = FileSystem.get(configuration);
        FileSystem fs = FileSystem.get(new URI("hdfs://hadoop1:9000"), configuration, "root"); # 需要读者写自己的

        //3.创建目录
        fs.mkdirs(new Path("//lxz/hdfs/first"));

        //4.关闭资源
        fs.close();
    }
}

 

三 在resources目录下创建log4j.properties文件

因为如果不配置log4j文件,则运行时会报错如下

 

 log4j配置如下

log4j.rootLogger=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - %m%n
log4j.appender.logfile=org.apache.log4j.FileAppender
log4j.appender.logfile.File=target/spring.log
log4j.appender.logfile.layout=org.apache.log4j.PatternLayout
log4j.appender.logfile.layout.ConversionPattern=%d %p [%c] - %m%n

 

四 运行程序,观察HDFS集群是否成功反应