物联网架构成长之路(48)-MinIO对象资源存储

0.前言
　　在开发物联网过程中，会遇到OTA固件升级等功能。对于服务器来说，其实就很简单的一个功能，无非就是一个上传文件，保存，下载文件的功能而已。在此之前，我也通过简单的文件系统实现文件上传下载。然后把路径保存到数据。也有人使用阿里的OSS来管理。但是今天要讲的就是搭建一个开源版本的OSS存储服务器。


1.安装MinIO
　　流程的话，参考官方文档，https://docs.min.io/cn/minio-quickstart-guide.html，由于使用go语言开发的，整个过程还是比较轻松简单的。这里我使用docker来运行MinIO。下面这个图，是我搭建的架构图。

　　这里，默认读者是已经安装过Docker和docker-compose的。下面提供docker-compose.yml
　　可以直接从这里下载：https://raw.githubusercontent.com/minio/minio/master/docs/orchestration/docker-compose/docker-compose.yaml

 1 version: '3.7'
 2 
 3 # starts 4 docker containers running minio server instances. Each
 4 # minio server's web interface will be accessible on the host at port
 5 # 9001 through 9004.
 6 services:
 7   minio1:
 8     image: minio/minio:RELEASE.2019-12-24T23-04-45Z
 9     volumes:
10       - data1-1:/data1
11       - data1-2:/data2
12     ports:
13       - "9001:9000"
14     environment:
15       MINIO_ACCESS_KEY: minio
16       MINIO_SECRET_KEY: minio123
17     command: server http://minio{1...4}/data{1...2}
18     healthcheck:
19       test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
20       interval: 30s
21       timeout: 20s
22       retries: 3
23 
24   minio2:
25     image: minio/minio:RELEASE.2019-12-24T23-04-45Z
26     volumes:
27       - data2-1:/data1
28       - data2-2:/data2
29     ports:
30       - "9002:9000"
31     environment:
32       MINIO_ACCESS_KEY: minio
33       MINIO_SECRET_KEY: minio123
34     command: server http://minio{1...4}/data{1...2}
35     healthcheck:
36       test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
37       interval: 30s
38       timeout: 20s
39       retries: 3
40 
41   minio3:
42     image: minio/minio:RELEASE.2019-12-24T23-04-45Z
43     volumes:
44       - data3-1:/data1
45       - data3-2:/data2
46     ports:
47       - "9003:9000"
48     environment:
49       MINIO_ACCESS_KEY: minio
50       MINIO_SECRET_KEY: minio123
51     command: server http://minio{1...4}/data{1...2}
52     healthcheck:
53       test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
54       interval: 30s
55       timeout: 20s
56       retries: 3
57 
58   minio4:
59     image: minio/minio:RELEASE.2019-12-24T23-04-45Z
60     volumes:
61       - data4-1:/data1
62       - data4-2:/data2
63     ports:
64       - "9004:9000"
65     environment:
66       MINIO_ACCESS_KEY: minio
67       MINIO_SECRET_KEY: minio123
68     command: server http://minio{1...4}/data{1...2}
69     healthcheck:
70       test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
71       interval: 30s
72       timeout: 20s
73       retries: 3
74 
75 ## By default this config uses default local driver,
76 ## For custom volumes replace with volume driver configuration.
77 volumes:
78   data1-1:
79   data1-2:
80   data2-1:
81   data2-2:
82   data3-1:
83   data3-2:
84   data4-1:
85   data4-2:

　　docker-compose up 启动服务

　　通过浏览器访问如下

 

2.Nginx负载均衡
　　从上面可以看到，是启动了4个服务，这里可以理解是有4台minio服务器。这里配置一台Nginx作为前端负载均衡，nginx配置如下:

 1 upstream minio-server{
 2         #默认负载均衡策略有四种
 3         #1. 默认不设置，采用轮询
 4         #2. least_conn 最少连接负载均衡
 5         #3. ip_hash 会话持久化负载均衡
 6         #4. server 127.0.0.1 weight=1 带权重负载均衡
 7         least_conn;
 8         #ip_hash;
 9 
10         server 127.0.0.1:9001 weight=10;
11         server 127.0.0.1:9002 weight=10;
12         server 127.0.0.1:9003 weight=10;
13         server 127.0.0.1:9004 weight=10;
14 }
15 
16 server {
17         listen 81 default_server;
18         listen [::]:81 default_server;
19 
20         server_name _;
21 
22         location / {
23                 proxy_set_header Host $http_host; #注意这里的Header一定要带到Minio，否则认证不通过
24                 add_header X-Host 'MinIO from $upstream_addr';
25                 proxy_pass http://minio-server;
26         }
27 }

　　配置nginx.conf后，reload配置，访问资源，可以根据X-Host字段判断，Nginx是有进行负载均衡，每次分配到不同的机器上。

 

 

3.MinIO使用
　　Web端访问，在Web端，创建一个Demo的Bucket。然后就可以web端上传下载文件。默认Bucket是有策略（Policy）的权限控制。
　　可以在Web端修改策略为ReadOnly，这样就可以通过链接URL直接访问资源。类似图床的做法。

 

4.Java Client(基于SpringBoot)
　　MinIO的客户端有很多种，也支持很多编程语言，官方有个mc工具https://docs.min.io/cn/minio-client-quickstart-guide.html
　　下面主要讲使用Java语言如何接入服务器。参考：https://docs.min.io/cn/java-client-quickstart-guide.html
　　完整代码如下:
pom.xml

 1 <?xml version="1.0" encoding="UTF-8"?>
 2 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 3     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
 4     <modelVersion>4.0.0</modelVersion>
 5     <parent>
 6         <groupId>org.springframework.boot</groupId>
 7         <artifactId>spring-boot-starter-parent</artifactId>
 8         <version>2.2.2.RELEASE</version>
 9         <relativePath/> <!-- lookup parent from repository -->
10     </parent>
11     <groupId>com.wunaozai.demo</groupId>
12     <artifactId>WunaozaiDemo</artifactId>
13     <version>0.0.1-SNAPSHOT</version>
14     <name>JWunaozaiDemo</name>
15     <description>所有测试例子</description>
16 
17     <properties>
18         <java.version>1.8</java.version>
19         <maven-jar-plugin.version>3.0.0</maven-jar-plugin.version>
20     </properties>
21 
22     <dependencies>
23         <dependency>
24             <groupId>org.springframework.boot</groupId>
25             <artifactId>spring-boot-starter</artifactId>
26         </dependency>
27         <dependency>
28             <groupId>org.springframework.boot</groupId>
29             <artifactId>spring-boot-starter-web</artifactId>
30         </dependency>
31         <dependency>
32             <groupId>org.springframework.boot</groupId>
33             <artifactId>spring-boot-configuration-processor</artifactId>
34             <optional>true</optional>
35         </dependency>
36         
37         <!-- MinIO Client -->
38         <dependency>
39             <groupId>io.minio</groupId>
40             <artifactId>minio</artifactId>
41             <version>6.0.11</version>
42         </dependency>
43         <!-- 文件上传 -->
44         <dependency>
45             <groupId>commons-fileupload</groupId>
46             <artifactId>commons-fileupload</artifactId>
47             <version>1.3.3</version>
48         </dependency>
49 
50         <dependency>
51             <groupId>org.springframework.boot</groupId>
52             <artifactId>spring-boot-starter-test</artifactId>
53             <scope>test</scope>
54         </dependency>
55     </dependencies>
56 
57     <build>
58         <plugins>
59             <plugin>
60                 <groupId>org.springframework.boot</groupId>
61                 <artifactId>spring-boot-maven-plugin</artifactId>
62             </plugin>
63         </plugins>
64     </build>
65 
66 </project>

 

application.properties

1 #MinIO
2 minio.url=http://172.16.23.125:81
3 minio.access-key=minio
4 minio.secret-key=minio123
5 minio.bucket-name=demo


MinIOProperties.java

 1 package com.wunaozai.demo.minio;
 2 
 3 import org.springframework.boot.context.properties.ConfigurationProperties;
 4 
 5 /**
 6  * 注入配置到properties
 7  * @author wunaozai
 8  * @Date 2019-12-26
 9  */
10 @ConfigurationProperties(prefix="minio")
11 public class MinIOProperties {
12     /**
13      * Minio 服务地址
14      */
15     private String url;
16     /**
17      * Access Key
18      */
19     private String accessKey;
20     /**
21      * Secret Key
22      */
23     private String secretKey;
24     /**
25      * Bucket
26      */
27     private String bucketName;
28     public String getUrl() {
29         return url;
30     }
31     
32     public void setUrl(String url) {
33         this.url = url;
34     }
35     
36     public String getAccessKey() {
37         return accessKey;
38     }
39     
40     public void setAccessKey(String accessKey) {
41         this.accessKey = accessKey;
42     }
43     
44     public String getSecretKey() {
45         return secretKey;
46     }
47     
48     public void setSecretKey(String secretKey) {
49         this.secretKey = secretKey;
50     }
51     
52     public String getBucketName() {
53         return bucketName;
54     }
55     
56     public void setBucketName(String bucketName) {
57         this.bucketName = bucketName;
58     }
59     
60 }


MinIOAutoConfiguration.java

 1 package com.wunaozai.demo.minio;
 2 
 3 import org.springframework.boot.context.properties.EnableConfigurationProperties;
 4 import org.springframework.context.annotation.Configuration;
 5 
 6 /**
 7  * 初始化MinioClient 并自动注入到spring容器(Bean)
 8  * @author wunaozai
 9  * @Date 2019-12-26
10  */
11 @Configuration
12 @EnableConfigurationProperties({MinIOProperties.class})
13 public class MinIOAutoConfiguration {
14     
15 }

 

MinIOTemplate.java

 1 package com.wunaozai.demo.minio;
 2 
 3 import java.io.InputStream;
 4 
 5 import org.springframework.beans.factory.InitializingBean;
 6 import org.springframework.beans.factory.annotation.Autowired;
 7 import org.springframework.stereotype.Component;
 8 import org.springframework.util.Assert;
 9 
10 import io.minio.MinioClient;
11 import io.minio.ObjectStat;
12 
13 /**
14  * 抽象模版类
15  * @author wunaozai
16  * @Date 2019-12-26
17  */
18 @Component
19 public class MinioTemplate implements InitializingBean {
20     
21     @Autowired
22     private MinIOProperties properties;
23     
24     private MinioClient client;
25     
26     public void printInfo() {
27         System.out.println(properties.getAccessKey());
28     }
29     
30     @Override
31     public void afterPropertiesSet() throws Exception {
32         Assert.hasText(properties.getUrl(), "Minio url 为空");
33         Assert.hasText(properties.getAccessKey(), "Minio accessKey为空");
34         Assert.hasText(properties.getSecretKey(), "Minio secretKey为空");
35         this.client = new MinioClient(
36                 properties.getUrl(), properties.getAccessKey(), properties.getSecretKey());
37         
38     }
39 
40     public boolean checkBucketName(String bucketName) throws Exception {
41         return client.bucketExists(bucketName);
42     }
43     public String getObjectURL(String objectName, Integer expires) throws Exception {
44         return client.presignedGetObject(properties.getBucketName(), objectName, expires);
45     }
46     public void saveObject(String objectName, InputStream stream, long size, 
47             String contentType) throws Exception {
48         client.putObject(properties.getBucketName(), 
49                 objectName, stream, size, null, null, contentType);
50     }
51     public ObjectStat getObjectInfo(String objectName) throws Exception {
52         return client.statObject(properties.getBucketName(), objectName);
53     }
54     public void removeObject(String objectName) throws Exception {
55         client.removeObject(properties.getBucketName(), objectName);
56     }
57 
58 }


MinIORestController.java

 1 package com.wunaozai.demo.minio;
 2 
 3 import java.io.InputStream;
 4 import java.util.Iterator;
 5 
 6 import javax.servlet.http.HttpServletRequest;
 7 
 8 import org.springframework.beans.factory.annotation.Autowired;
 9 import org.springframework.web.bind.annotation.RequestMapping;
10 import org.springframework.web.bind.annotation.RestController;
11 import org.springframework.web.multipart.MultipartFile;
12 import org.springframework.web.multipart.MultipartHttpServletRequest;
13 import org.springframework.web.multipart.commons.CommonsMultipartResolver;
14 
15 import io.minio.ObjectStat;
16 
17 @RestController
18 @RequestMapping(value="/minio")
19 public class MinIORestController {
20     
21     
22     @Autowired
23     private MinioTemplate minioTemplate;
24     
25     @RequestMapping(value="/checkbucketname")
26     public String checkBucketName(String name) throws Exception {
27         return minioTemplate.checkBucketName(name) + "";
28     }
29     @RequestMapping(value="/get")
30     public String getObjectURL(String filename) throws Exception {
31         return minioTemplate.getObjectURL(filename, 2000);
32     }
33     @RequestMapping(value="/info")
34     public String getObjectInfo(String filename) throws Exception {
35         ObjectStat stat = minioTemplate.getObjectInfo(filename);
36         return stat.bucketName() + "#" + stat.name();
37     }
38     @RequestMapping(value="/put")
39     public String putObject(HttpServletRequest request) {
40         CommonsMultipartResolver resolver = new CommonsMultipartResolver(request.getSession().getServletContext());
41         if(!resolver.isMultipart(request)) {
42             return "error";
43         }
44         MultipartHttpServletRequest multiRequest = (MultipartHttpServletRequest) request;
45         String filename = multiRequest.getParameter("filename");
46         Iterator<String> it = multiRequest.getFileNames();
47         while(it.hasNext()) {
48             try {
49                 MultipartFile file = multiRequest.getFile(it.next());
50                 if(filename == null || filename.equals("")) {
51                     filename = file.getOriginalFilename();
52                 }
53                 long size = file.getSize();
54                 String contentType = file.getContentType();
55                 InputStream input = file.getInputStream();
56                 minioTemplate.saveObject(filename, input, size, contentType);
57                 return "ok";
58             } catch (Exception e) {
59                 e.printStackTrace();
60             }
61         }
62         return "error";
63         
64     }
65 }

 

　　基于上述代码，可以可以直接集成到项目里面。通过PostMan请求结果如下

 

参考资料：
　　https://docs.min.io/cn/java-client-quickstart-guide.html
　　https://github.com/aboullaite/minio-starter
　　https://blog.csdn.net/huangbaoling66/article/details/90179736
　　https://www.cnblogs.com/DarrenChan/p/8967412.html

本文地址：https://www.cnblogs.com/wunaozai/p/12103949.html
本系列目录： https://www.cnblogs.com/wunaozai/p/8067577.html
个人主页：https://www.wunaozai.com/

posted @ 2019-12-27 17:11  无脑仔的小明  阅读(2001)  评论(0)  编辑  收藏  举报