Hadoop-09-HDFS集群 JavaClient 代码上手实战！详细附代码 安装依赖 上传下载文件 扫描列表 PUT GET 进度条显示 原创

章节内容

上一节完成：

• HDFS的集群启动
• HDFS的命令行操作
• HDFS 上传下载移动重命名等操作

背景介绍

这里是三台公网云服务器，每台 2C4G，搭建一个Hadoop的学习环境，供我学习。
之前已经在 VM 虚拟机上搭建过一次，但是没留下笔记，这次趁着前几天薅羊毛的3台机器，赶紧尝试在公网上搭建体验一下。

注意，如果你和我一样，打算用公网部署，那一定要做好防火墙策略，避免不必要的麻烦！！！
请大家都以学习为目的，也请不要对我的服务进行嗅探或者攻击！！！

但是有一台公网服务器我还运行着别的服务，比如前几天发的：autodl-keeper 自己写的小工具，防止AutoDL机器过期的。还跑着别的Web服务，所以只能挤出一台 2C2G 的机器。那我的配置如下了：

• 2C4G 编号 h121
• 2C4G 编号 h122
• 2C2G 编号 h123

新建工程

这里使用IDEA新建一个Maven工程即可！

POM

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>hadoop-demo</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <!-- Hadoop Dependencies -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.9.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.9.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-core</artifactId>
            <version>2.9.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-common</artifactId>
            <version>2.9.0</version>
        </dependency>
    </dependencies>

</project>

创建文件

package icu.wzk.demo01;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

import java.io.IOException;

public class Main {
    public static void main(String[] args) throws IOException {
        // 创建文件
        mkdirs();
    }

    public static void mkdirs() throws IOException {
        Configuration configuration = new Configuration();
        configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
        FileSystem fileSystem = FileSystem.get(configuration);
        fileSystem.mkdirs(new Path("/wzk-test"));
        fileSystem.close();
    }
}

我们对应的查看HDFS集群上的目录，是否一致

上传文件

public static void upload() throws IOException {
    Configuration configuration = new Configuration();
    configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
    FileSystem fileSystem = FileSystem.get(configuration);
    Path filePath = new Path("wzk01.txt");
    Path toFilePath = new Path("/wzk-test/wzk01.txt");
    fileSystem.copyFromLocalFile(filePath, toFilePath);
    fileSystem.close();
}

在HDFS中查看对应的文件

下载文件

public static void download() throws IOException {
    Configuration configuration = new Configuration();
    configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
    FileSystem fileSystem = FileSystem.get(configuration);
    Path filePath = new Path("/wzk-test/wzk01.txt");
    Path toFilePath = new Path("wzk01-01.txt");
    fileSystem.copyToLocalFile(filePath, toFilePath);
    fileSystem.close();
}

删除文件

public static void delete() throws IOException {
    Configuration configuration = new Configuration();
    configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
    FileSystem fileSystem = FileSystem.get(configuration);
    fileSystem.delete(new Path("/wzk-test"), true);
    fileSystem.close();
}

展示列表

public static void listList() throws IOException {
    Configuration configuration = new Configuration();
    configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
    FileSystem fileSystem = FileSystem.get(configuration);
    RemoteIterator<LocatedFileStatus> listFiles = fileSystem.listFiles(new Path("/"), true);
    while (listFiles.hasNext()) {
        LocatedFileStatus status = listFiles.next();
        System.out.println("文件名字: " + status.getPath().getName());
        System.out.println("文件长度: " + status.getLen());
        System.out.println("文件块大小: " + status.getBlockSize());
        System.out.println("权限: " + status.getPermission());
        System.out.println("分组: " + status.getGroup());
        BlockLocation[] blockLocations = status.getBlockLocations();
        for (BlockLocation blockLocation : blockLocations) {
            String[] hosts = blockLocation.getHosts();
            for (String host : hosts) {
                System.out.println(host);
            }
        }
        System.out.println("==========================");
    }
    fileSystem.close();
}

扫描路径

public static void listStatus() throws IOException {
    Configuration configuration = new Configuration();
    configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
    FileSystem fileSystem = FileSystem.get(configuration);
    FileStatus[] listStatus = fileSystem.listStatus(new Path("/"));
    for (FileStatus fileStatus : listStatus) {
        if (fileStatus.isFile()) {
            System.out.println("文件: " + fileStatus.getPath().getName());
        } else {
            System.out.println("文件夹: " + fileStatus.getPath().getName());
        }
    }
    fileSystem.close();
}

PUT 操作

public static void putFile() throws IOException {
    Configuration configuration = new Configuration();
    configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
    FileSystem fileSystem = FileSystem.get(configuration);
    try (FileInputStream fis= new FileInputStream("wzk02.txt")) {
        FSDataOutputStream fos = fileSystem.create(new Path("/wzk02_io.txt"));
        IOUtils.copyBytes(fis, fos, configuration);
        IOUtils.closeStream(fos);
        IOUtils.closeStream(fis);
        fileSystem.close();
    }
}

GET 操作

public static void getFile() throws IOException {
    Configuration configuration = new Configuration();
    configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
    FileSystem fileSystem = FileSystem.get(configuration);
    try (FileOutputStream fos = new FileOutputStream("wzk02_io_get.txt")) {
        FSDataInputStream fis = fileSystem.open(new Path("/wzk02_io.txt"));
        IOUtils.copyBytes(fis, fos, configuration);
        IOUtils.closeStream(fos);
        IOUtils.closeStream(fis);
        fileSystem.close();
    }
}

Seek操作

public static void seek() throws IOException {
    Configuration configuration = new Configuration();
    configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
    FileSystem fileSystem = FileSystem.get(configuration);
    FSDataInputStream in = null;
    try {
        in  = fileSystem.open(new Path("/wzk02_io.txt"));
        IOUtils.copyBytes(in, System.out, 4096, false);
        in.seek(0);
        IOUtils.copyBytes(in, System.out, 4096, false);
    } finally {
        IOUtils.closeStream(in);
    }
}

进度显示

public static void uploadProgress() throws IOException {
    Configuration configuration = new Configuration();
    configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
    FileSystem fileSystem = FileSystem.get(configuration);
    try (FileInputStream fis = new FileInputStream("music.mp3")) {
        FSDataOutputStream fos = fileSystem.create(new Path("/wzk/music.mp3"),
                () -> System.out.print("="));
        IOUtils.copyBytes(fis, fos, configuration);
        IOUtils.closeStream(fos);
        IOUtils.closeStream(fis);
        fileSystem.close();
        System.out.println("done!");
    }
}