FastDFS分布式文件系统环境搭建和代码测试
FastDFS分布式文件系统环境搭建和代码测试
待配置的机器：
192.168.44.10 192.168.44.11 192.168.44.12 192.168.44.13
注意：关闭linux防火墙，或者开放FastDFS所需的一些列端口
一、安装必要的工具和依赖
1、工具
yum install lrzsz wget vim unzip net-tools -y
2、依赖
yum install gcc perl openssl openssl-devel pcre pcre-devel zlib zlib-devel libevent libevent-devel -y
二、安装FastDFS
1、安装包下载地址
码云地址：https://gitee.com/fastdfs100
下载FastDFS、libfastcommon、fastdfs-nginx-module(可选)、fastdfs-client-java(可选)
使用rz -y 上传文件，使用sz下载文件，这个工具在lrzsz安装，有这个工具后，用拖拽也可以。
2、安装libfastcommon
libfastcommon 库是 FastDFS 文件系统运行需要的公共 C 语言函数库
解压：tar -zxvf fastdfs100-libfastcommon-V1.0.51.tar.gz
如果是zip包使用：unzip fastdfs100-libfastcommon-V1.0.51.zip
进入目录，编译：./make.sh
编译没报错可以安装：./make.sh install
3、安装FastDFS
解压：tar -zxvf fastdfs100-fastdfs-V6.07.tar.gz
或 unzip fastdfs100-fastdfs-V6.07.zip
进入目录，编译：./make.sh
编译没报错可以安装：./make.sh install
所有编译出来的文件存放在/usr/bin****目录下
所有配置文件存放在/etc/fdfs****目录下
通过ls /usr/bin/fdfs* 可以查看所有fastDFS的命令
三、配置FastDFS
1、将/etc/fdfs下的配置文件样例全部改名，去掉sample后缀
mv client.conf.sample client.conf
mv storage.conf.sample storage.conf
mv storage_ids.conf.sample storage_ids.conf
mv tracker.conf.sample tracker.conf
2、将安装目录conf下的http.conf和拷贝到/etc/fdfs下
cp http.conf /etc/fdfs/
cp mime.types /etc/fdfs/
3、分组
group1：192.168.44.10 192.168.44.11
group2： 192.168.44.12 192.168.44.13
tracker server: 192.168.44.10 192.168.44.12
storage server: 全部机器
group1和group2的数据在各自的组内自动复制，做冗余备份。
group1和group2之间数据不重复，只为提高并发和容量。
4、创建日志目录和数据目录
mkdir -p /opt/fastdfs/tracker #tracker的日志目录
mkdir -p /opt/fastdfs/storage #storage的日志目录
mkdir -p /opt/fastdfs/storage/files #真正存放文件的目录
5、修改tracker.conf
192.168.44.10
base_path = /opt/fastdfs/tracker
store_group = group1
192.168.44.12
base_path = /opt/fastdfs/tracker
store_group = group2
store_group参数与store_lookup参数相关
注意：FastDFS默认是带有负载均衡策略的可以在tracker的2台机器中修改tracker.conf文件
store_lookup=10 随机存放策略
1 指定组
2 选择磁盘空间的优先存放 默认值修改后重启服务
fdfs_trackerd /etc/fdfs/tracker.conf restart
6、修改storage.conf
# group1或group2，看分组配置
group_name = group1
base_path = /opt/fastdfs/storage
store_path0 = /opt/fastdfs/storage/files
tracker_server = 192.168.44.10:22122
tracker_server = 192.168.44.12:22122
7、启动tracker server
192.168.44.10和192.168.44.12
fdfs_trackerd /etc/fdfs/tracker.conf start
ps -ef | grep fdfs
root 1753 1 0 11:30 ? 00:00:00 fdfs_trackerd /etc/fdfs/tracker.conf
8、启动storage server
fdfs_storaged /etc/fdfs/storage.conf start
需要通过ps -ef | grep fdfs检查是否启动。
/opt/fastdfs/storage/files/data 这个目录下是数据存放位置，data目录是FastDFS自动创建，这个目录下有00~FF个目录
9、查看storage注册情况
fdfs_monitor /etc/fdfs/storage.conf
10、重启FastDFS
重启tracker
fdfs_trackerd /etc/fdfs/tracker.conf restart
重启storage
fdfs_storaged /etc/fdfs/storage.conf restart
11、关闭FastDFS
fdfs_trackerd /etc/fdfs/tracker.conf stop
fdfs_storaged /etc/fdfs/storage.conf stop
不建议在生产环境使用 kill -9 强制关闭，因为可能会导致文件信息不同步问题
四、环境测试
使用FastDFS自带的测试工具
1、修改client.conf配置文件，主要修改两个配置：
base_path=/opt/fastdfs/client
tracker_server=192.168.44.10:22122
2、测试
执行上传命令
fdfs_test /etc/fdfs/client.conf upload /root/test.txt
删除测试
fdfs_delete_file /etc/fdfs/client.conf group1/要删除的文件路径
注意：
没有搭建集群默认只有一个组group1
后缀名包含-m的为属性文件(meta)
五、代码测试
使用Springboot进行代码测试
1、依赖
<dependency>
<groupId>com.github.tobato</groupId>
<artifactId>fastdfs-client</artifactId>
<version>1.27.2</version>
</dependency>
2、service
property外置前缀upload到application.yml里
@ConfigurationProperties(prefix = "upload")
@Data
public class UploadProperties {
private String baseUrl;
private List<String> allowTypes;
}
UploadService
package com.home.system.service;
import com.github.tobato.fastdfs.domain.fdfs.FileInfo;
import com.github.tobato.fastdfs.domain.fdfs.StorePath;
import com.github.tobato.fastdfs.service.FastFileStorageClient;
import com.home.system.config.UploadProperties;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile;
import javax.annotation.Resource;
import javax.imageio.ImageIO;
import javax.servlet.http.HttpServletResponse;
import java.awt.image.BufferedImage;
import java.io.Closeable;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.HashSet;
/**
* @Author: xu.dm
* @Date: 2021/6/16 14:44
* @Version: 1.0
* @Description: TODO
**/
@Slf4j
@EnableConfigurationProperties(UploadProperties.class)
@Service
public class UploadService {
@Resource
private FastFileStorageClient storageClient;
@Resource
private UploadProperties properties;
public String uploadImage(MultipartFile file) {
// 1、校验文件类型
String contentType = file.getContentType();
if (!properties.getAllowTypes().contains(contentType)) {
throw new RuntimeException("文件类型不支持");
}
// 2、校验文件内容
try {
BufferedImage image = ImageIO.read(file.getInputStream());
if (image == null || image.getWidth() == 0 || image.getHeight() == 0) {
throw new RuntimeException("上传文件有问题");
}
} catch (IOException e) {
log.error("校验文件内容失败....{}", e);
throw new RuntimeException("校验文件内容失败" + e.getMessage());
}
try {
// 3、上传到FastDFS
// 3.1、获取扩展名
String extension = StringUtils.substringAfterLast(file.getOriginalFilename(), ".");
// 3.2、上传
StorePath storePath = storageClient.uploadFile(file.getInputStream(), file.getSize(), extension, new HashSet<>());
System.out.println(storePath.getGroup());
System.out.println(storePath.getPath());
System.out.println(storePath);
// 返回路径
return properties.getBaseUrl() + storePath.getFullPath();
} catch (IOException e) {
log.error("【文件上传】上传文件失败！...." + e);
throw new RuntimeException("【文件上传】上传文件失败！" + e.getMessage());
}
}
public String uploadFile(MultipartFile file) {
try {
// 3、上传到FastDFS
// 3.1、获取扩展名
String extension = StringUtils.substringAfterLast(file.getOriginalFilename(), ".");
// 3.2、上传
StorePath storePath = storageClient.uploadFile(file.getInputStream(), file.getSize(), extension, new HashSet<>());
System.out.println(storePath.getGroup());
System.out.println(storePath.getPath());
System.out.println(storePath);
// 返回路径
return properties.getBaseUrl() + storePath.getFullPath();
} catch (IOException e) {
log.error("【文件上传】上传文件失败！...." + e);
throw new RuntimeException("【文件上传】上传文件失败！" + e.getMessage());
}
}
public void downloadFile(HttpServletResponse response, String groupName, String path) {
FileInfo fileInfo = null;
try {
fileInfo = storageClient.queryFileInfo(groupName, path);
}catch (Exception e){
throw new RuntimeException("文件不存在");
}
if(fileInfo == null) {
throw new RuntimeException("文件不存在");
}
InputStream inputStream = storageClient.downloadFile(groupName, path, ins -> ins);
String filename = path.replace("/", ".");
response.setContentType("application/octet-stream;charset=UTF-8");
response.setHeader("Content-Disposition", "attachment; filename="+filename);
try {
streamCopy(inputStream,response.getOutputStream());
response.flushBuffer();
} catch (IOException e) {
closeQuietly(inputStream);
e.printStackTrace();
throw new RuntimeException("下载文件失败！");
}
}
public void deleteFile(String groupName, String path) {
storageClient.deleteFile(groupName, path);
}
private void streamCopy(InputStream inp, OutputStream out) throws IOException {
byte[] buff = new byte[4096];
int count;
while ((count = inp.read(buff)) != -1) {
if (count > 0) {
out.write(buff, 0, count);
}
}
}
public static void closeQuietly( final Closeable closeable ) {
// no need to log a NullPointerException here
if(closeable == null) {
return;
}
try {
closeable.close();
} catch ( Exception e ) {
log.error( "Unable to close resource: " + e,
e );
}
}
}
3、controller
package com.home.system.controller;
import com.home.system.service.UploadService;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.multipart.MultipartFile;
import javax.annotation.Resource;
import javax.servlet.http.HttpServletResponse;
import java.util.HashMap;
import java.util.Map;
/**
* @Author: xu.dm
* @Date: 2021/6/16 14:50
* @Version: 1.0
* @Description: TODO
**/
@Controller
public class UploadController {
@Resource
private UploadService uploadService;
@GetMapping("/upload")
public String upload() {
return "upload";
}
@GetMapping("/fileDelete")
public String fileDelete() {
return "fileDelete";
}
@GetMapping("/fileDownload")
public String fileDownload() {
return "download";
}
@PostMapping("file/download")
public void doDownload(HttpServletResponse response, String groupName, String path){
uploadService.downloadFile(response, groupName, path);
}
@PostMapping("file/delete")
@ResponseBody
public String doDelete(String groupName, String path){
uploadService.deleteFile(groupName, path);
return "删除成功";
}
@RequestMapping("upload/doUpload")
@ResponseBody
public Map<String,Object> doUpload(MultipartFile file){
System.out.println(file.getOriginalFilename());
Map<String, Object> upload =new HashMap<>();
String path = this.uploadService.uploadImage(file);
upload.put("path",path);
return upload;
}
@RequestMapping("upload/doFileUpload")
@ResponseBody
public Map<String,Object> doFileUpload(MultipartFile file){
System.out.println(file.getOriginalFilename());
Map<String, Object> upload =new HashMap<>();
String path = this.uploadService.uploadFile(file);
upload.put("path",path);
return upload;
}
}
4、application.yml文件配置
server:
port: 8080
tomcat:
accept-count: 3
max-connections: 6
threads:
max: 10
basedir: ./target/
logging:
file:
# 一旦日志文件大于设定值，这个日志文件就会被加上日期归档，原文清空重新开始记录日志
name: ./target/myBoot.log
logback:
rollingpolicy:
max-file-size: 20MB
max-history: 30
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
servlet:
multipart:
max-request-size: 500MB
max-file-size: 500MB
fdfs:
connect-timeout: 60000
so-timeout: 120000
tracker-list:
- 192.168.44.10:22122
- 192.168.44.12:22122
upload:
base-url: http://192.168.1.105/ #换成你的ip
allow-types:
- image/jpeg
- image/png
- image/bmp
