Hadoop-09-HDFS集群 JavaClient 代码上手实战!详细附代码 安装依赖 上传下载文件 扫描列表 PUT GET 进度条显示 原创
章节内容
上一节完成:
- HDFS的集群启动
- HDFS的命令行操作
- HDFS 上传下载移动重命名等操作
背景介绍
这里是三台公网云服务器,每台 2C4G,搭建一个Hadoop的学习环境,供我学习。
之前已经在 VM 虚拟机上搭建过一次,但是没留下笔记,这次趁着前几天薅羊毛的3台机器,赶紧尝试在公网上搭建体验一下。
注意,如果你和我一样,打算用公网部署,那一定要做好防火墙策略,避免不必要的麻烦!!!
请大家都以学习为目的,也请不要对我的服务进行嗅探或者攻击!!!
但是有一台公网服务器我还运行着别的服务,比如前几天发的:autodl-keeper 自己写的小工具,防止AutoDL机器过期的。还跑着别的Web服务,所以只能挤出一台 2C2G 的机器。那我的配置如下了:
- 2C4G 编号 h121
- 2C4G 编号 h122
- 2C2G 编号 h123
新建工程
这里使用IDEA新建一个Maven工程即可!
POM
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>hadoop-demo</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<!-- Hadoop Dependencies -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-common</artifactId>
<version>2.9.0</version>
</dependency>
</dependencies>
</project>
创建文件
package icu.wzk.demo01;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import java.io.IOException;
public class Main {
public static void main(String[] args) throws IOException {
// 创建文件
mkdirs();
}
public static void mkdirs() throws IOException {
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
FileSystem fileSystem = FileSystem.get(configuration);
fileSystem.mkdirs(new Path("/wzk-test"));
fileSystem.close();
}
}
我们对应的查看HDFS集群上的目录,是否一致
上传文件
public static void upload() throws IOException {
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
FileSystem fileSystem = FileSystem.get(configuration);
Path filePath = new Path("wzk01.txt");
Path toFilePath = new Path("/wzk-test/wzk01.txt");
fileSystem.copyFromLocalFile(filePath, toFilePath);
fileSystem.close();
}
在HDFS中查看对应的文件
下载文件
public static void download() throws IOException {
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
FileSystem fileSystem = FileSystem.get(configuration);
Path filePath = new Path("/wzk-test/wzk01.txt");
Path toFilePath = new Path("wzk01-01.txt");
fileSystem.copyToLocalFile(filePath, toFilePath);
fileSystem.close();
}
删除文件
public static void delete() throws IOException {
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
FileSystem fileSystem = FileSystem.get(configuration);
fileSystem.delete(new Path("/wzk-test"), true);
fileSystem.close();
}
展示列表
public static void listList() throws IOException {
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
FileSystem fileSystem = FileSystem.get(configuration);
RemoteIterator<LocatedFileStatus> listFiles = fileSystem.listFiles(new Path("/"), true);
while (listFiles.hasNext()) {
LocatedFileStatus status = listFiles.next();
System.out.println("文件名字: " + status.getPath().getName());
System.out.println("文件长度: " + status.getLen());
System.out.println("文件块大小: " + status.getBlockSize());
System.out.println("权限: " + status.getPermission());
System.out.println("分组: " + status.getGroup());
BlockLocation[] blockLocations = status.getBlockLocations();
for (BlockLocation blockLocation : blockLocations) {
String[] hosts = blockLocation.getHosts();
for (String host : hosts) {
System.out.println(host);
}
}
System.out.println("==========================");
}
fileSystem.close();
}
扫描路径
public static void listStatus() throws IOException {
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
FileSystem fileSystem = FileSystem.get(configuration);
FileStatus[] listStatus = fileSystem.listStatus(new Path("/"));
for (FileStatus fileStatus : listStatus) {
if (fileStatus.isFile()) {
System.out.println("文件: " + fileStatus.getPath().getName());
} else {
System.out.println("文件夹: " + fileStatus.getPath().getName());
}
}
fileSystem.close();
}
PUT 操作
public static void putFile() throws IOException {
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
FileSystem fileSystem = FileSystem.get(configuration);
try (FileInputStream fis= new FileInputStream("wzk02.txt")) {
FSDataOutputStream fos = fileSystem.create(new Path("/wzk02_io.txt"));
IOUtils.copyBytes(fis, fos, configuration);
IOUtils.closeStream(fos);
IOUtils.closeStream(fis);
fileSystem.close();
}
}
GET 操作
public static void getFile() throws IOException {
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
FileSystem fileSystem = FileSystem.get(configuration);
try (FileOutputStream fos = new FileOutputStream("wzk02_io_get.txt")) {
FSDataInputStream fis = fileSystem.open(new Path("/wzk02_io.txt"));
IOUtils.copyBytes(fis, fos, configuration);
IOUtils.closeStream(fos);
IOUtils.closeStream(fis);
fileSystem.close();
}
}
Seek操作
public static void seek() throws IOException {
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
FileSystem fileSystem = FileSystem.get(configuration);
FSDataInputStream in = null;
try {
in = fileSystem.open(new Path("/wzk02_io.txt"));
IOUtils.copyBytes(in, System.out, 4096, false);
in.seek(0);
IOUtils.copyBytes(in, System.out, 4096, false);
} finally {
IOUtils.closeStream(in);
}
}
进度显示
public static void uploadProgress() throws IOException {
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://h121.wzk.icu:9000");
FileSystem fileSystem = FileSystem.get(configuration);
try (FileInputStream fis = new FileInputStream("music.mp3")) {
FSDataOutputStream fos = fileSystem.create(new Path("/wzk/music.mp3"),
() -> System.out.print("="));
IOUtils.copyBytes(fis, fos, configuration);
IOUtils.closeStream(fos);
IOUtils.closeStream(fis);
fileSystem.close();
System.out.println("done!");
}
}
· 分享4款.NET开源、免费、实用的商城系统
· 全程不用写代码,我用AI程序员写了一个飞机大战
· MongoDB 8.0这个新功能碉堡了,比商业数据库还牛
· 白话解读 Dapr 1.15:你的「微服务管家」又秀新绝活了
· 上周热点回顾(2.24-3.2)