MinIO安装从入门到精通
一、MinIO概述
MinIO 是在 Apache License v2.0 下发布的高性能对象存储。它是与 Amazon S3 云存储服务兼容的 API。使用 MinIO 构建 用于机器学习、分析和应用的高性能基础设施数据工作负载。MinIO 从根本上与众不同,专为企业和私有云设计。MinIO生产部署涵盖了全球。MinIO是全球使用最多和下载量最大的对象存储服务系统，还是全世界增长最快的对象存储系统。
官方文档：http://docs.minio.org.cn/docs/(中文文档)
主要有以下几个特点：
二、MinIO单机部署(docker 单节点单驱动)
MinIO强烈推荐生产集群由 minimum 4 个组成 minio server 节点在服务器池。
2.1 准备工作
$ mkdir -p /data/minio/data
$ mkdir -p /data/minio/config
$ docker run -d \
-p 9000:9000 \
-p 9090:9090 \
--name minio \
-v /data/data:/data \
-v /data/minio/config:/root/.minio \
-e "MINIO_ROOT_USER=minioadmin" \
-e "MINIO_ROOT_PASSWORD=minioadmin" \
quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z server /data --console-address ":9090
参数说明：
- docker run启动 MinIO 容器
- -p将本地端口绑定到容器端口
- -name为容器创建一个名称
- 9000表示MinIO服务地址，其上传调用的就是这个服务地址
- 9090表示MinIO的Web Console地址，Console监听的是一个动态的端口, 可以通过 --console-address ":port" 指定静态端口
- /mnt/data/minio/data:/data表示将MiniIO的数据挂载到宿主机上
- /mnt/data/minio/config:/root/.minio表示将MiniIO的配置文件挂载到宿主机上
- MINIO_ROOT_USER=minioadmin表示MinIO部署的root用户的用户名(accessKey)，不写默认的用户名就是minioadmin
- MINIO_ROOT_PASSWORD=minioadmin表示MinIO部署的root用户的密码(secretKey)，不写默认的密码就是minioadmin
2.2 部署
提前拉取镜像，docker pull quay.io/minio/minio:latest 这里采用最新稳定版镜像。
$ docker run \
> -p 9000:9000 \
> -p 9090:9090 \
> --name minio \
> -v /data/data:/data \
> -v /data/minio/config:/root/.minio \
> -e "MINIO_ROOT_USER=minioadmin" \
> -e "MINIO_ROOT_PASSWORD=minioadmin" \
> quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z server /data --console-address ":9090"
Unable to find image 'quay.io/minio/minio:latest' locally
latest: Pulling from minio/minio
d5d2e87c6892: Pull complete
008dba906bf6: Pull complete
e632dfd7f0e0: Pull complete
bb2a22a09061: Pull complete
19b7bd2519a4: Pull complete
a2d565af0f6d: Pull complete
f0067029ce19: Downloading [============> ] 10.65MB/42.53MB
安装成功后如下：
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1aa0bf931300 quay.io/minio/minio "/usr/bin/docker-ent…" 6 seconds ago Up 5 seconds 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp minio
[root@xiaohezi ~]#
2.3 登录Web Console
浏览器中输入http://ip:9090/，用户名密码登录系统。
三、MinIO单机部署（二进制单节点单驱动）
3.1 创建数据目录
$ mkdir /data/minio/data -p
3.2 下载二进制包
$ wget https://dl.min.io/server/minio/release/linux-amd64/minio
$ chmod +x minio
$ mv minio /usr/local/bin/
$ minio --version
minio version RELEASE.2022-09-25T15-44-53Z (commit-id=877bd95fa312c5282c3aa0b73c75af43af9c5914)
Runtime: go1.18.6 linux/amd64
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Copyright: 2015-2022 MinIO, Inc.
3.3 设置环境变量
配置用户名密码
$ export MINIO_ROOT_USER=minioadmin
$ export MINIO_ROOT_PASSWORD=minioadmin
3.4 创建服务启动文件（可选）
vim /usr/lib/systemd/system/minio.service
[Unit]
Description=Minio service
Documentation=https://docs.minio.io/
[Service]
WorkingDirectory=/data/minio/run/
ExecStart=/usr/local/bin/minio --console-address ":9090" /data/minio/data
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
3.5 启动
$ minio server --console-address ":9090" /data/minio/data
或者:
$ systemctl daemon-reload
$ systemctl start minio.service && systemctl status minio.service
$ systemctl enable minio.service
$ minio server --console-address ":9090" /data/minio/data
WARNING: Detected Linux kernel version older than 4.0.0 release, there are some known potential performance problems with this kernel version. MinIO recommends a minimum of 4.x.x linux kernel version for best performance
Formatting 1st pool, 1 set(s), 1 drives per set.
WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable.
WARNING: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables
MinIO Object Storage Server
Copyright: 2015-2022 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: RELEASE.2022-09-25T15-44-53Z (go1.18.6 linux/amd64)
Status: 1 Online, 0 Offline.
API: http://192.168.0.4:9000 http://172.17.0.1:9000 http://127.0.0.1:9000
RootUser: minioadmin
RootPass: minioadmin
Console: http://192.168.0.4:9090 http://172.17.0.1:9090 http://127.0.0.1:9090
RootUser: minioadmin
RootPass: minioadmin
Command-line: https://docs.min.io/docs/minio-client-quickstart-guide
$ mc alias set myminio http://192.168.0.4:9000 minioadmin minioadmin
Documentation: https://docs.min.io
通过上面可以看到用户名密码为minioadmin，默认启动的API端口是9000，API 默认端口可通过--address IP:PORT来指定；还有一个Web Console的端口，并且Console监听的是一个动态的端口, 可以通过 --console-address ":port" 指定静态端口。
默认的配置目录是{HOME}/.minio，可以通过 --config-dir命令自定义配置目录
$ minio server -config-dir /data/minio/config --console-address ":9090" --address ":9000" /data/minio/data
后台启动
nohup minio server -config-dir /data/minio/config --console-address ":9090" --address ":9000" /data/minio/data >/data/minio/minio.log 2>&1 &
四、MinIO单机部署(docker-compose单节点单驱动)
4.1 准备工作
编写docker-compose-minio.yaml文件，内容如下：
version: '3.7'
services:
minio:
image: quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z
container_name: minio
hostname: minio
restart: always
command: server --console-address ":9090" /data
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
TZ: Asia/Shanghai
volumes:
- /data/minio/data:/data
- /data/minio/config:/root/.minio
ports:
- "9000:9000"
- "9090:9090"
networks:
- pk_net
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
networks:
pk_net:
external: true
自定义docker网络
### 预先创建一个自定义的网络pk_net,此处的10.139可以自定义，不冲突即可
$ sudo docker network create --driver bridge --subnet 10.139.0.0/16 --gateway 10.139.0.1 pk_net
参数说明：healthcheck表示健康探测 server --console-address ":9090" /data 表示指定的minio服务下面挂载的目标磁盘为/data，并且指定Web Console的端口
4.2 部署
$ docker-compose -f docker-compose-minio.yaml up -d
五、MinIO单机部署(docker-compose单节点多驱动)
5.1 准备工作
挂载的磁盘增加到四个,单机版部署也可挂载多个磁盘，单个服务挂载超过(等于)4个磁盘，自动启动纠删码模式，可以预防磁盘损坏的情况下，导致文件不丢失。
调整docker-compose-minio.yaml文件，内容如下：
version: '3.7'
services:
minio:
image: quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z
container_name: minio
hostname: minio
restart: always
command: server --console-address ":9090" http://minio/data{1...4}
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
TZ: Asia/Shanghai
volumes:
- /data/minio/data1:/data1
- /data/minio/data2:/data2
- /data/minio/data3:/data3
- /data/minio/data4:/data4
- /data/minio/config:/root/.minio
ports:
- "9000:9000"
- "9090:9090"
networks:
- pk_net
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
networks:
pk_net:
external: true
参数说明：healthcheck表示健康探测 http://minio/data{1...4} 表示指定的minio服务下面挂载的目标磁盘为/data1、/data2、/data3和/data4 单机版部署也可挂载多个磁盘，单个服务挂载超过(等于)4个磁盘，自动启动纠删码模式，可以预防磁盘损坏的情况下，导致文件不丢失。
5.2 部署
$ docker-compose -f docker-compose-minio.yaml up -d
六、MinIO分布式部署(docker-compose模拟多节点多驱动)
分布式 Minio 可以让你将多块硬盘（甚至是在不同的机器上）组成一个对象存储服务。由于硬盘分布在不同的节点上，分布式 Minio 避免了单点故障。
数据保护：分布式 Minio 采用纠删码来防范多个节点宕机和位衰减 bit rot。分布式 Minio 至少需要四块硬盘，使用分布式 Minio 自动引入了纠删码功能。MinIO 使用纠删码 erasure code 和校验和 checksum 来保护数据免受硬件故障和无声数据损坏。即便您丢失一半数量 (N/2) 的硬盘，依然可以使用。纠删码是一种恢复丢失和损坏数据的数学算法，MinIO 采用 Reed-Solomon code 将对象拆分成 N/2 数据和 N/2 奇偶校验块。这就意味着如果是 12 块硬盘，一个对象会被分成 6 个数据块、6 个奇偶校验块，你可以丢失任意 6 块（不管其是否存放的数据还是奇偶校验块），你仍可以从剩下的盘中的数据进行恢复。
高可用：单机节点存在单点故障，相反，如果是一个有 N 块硬盘的分布式 MinIO，只要有 N/2 块硬盘在线，你的数据就是安全的。不过你需要有 N/2 + 1 个硬盘来创建新的对象。例如，一个 16 个节点的 MinIO 集群，每个节点 16 块硬盘，就算 8 台服务器宕机，这个集群仍然是可读的，不过需要有9台服务器才能写数据
一致性：MinIO 在分布式和单机模式下，所有读写操作都严格遵守 read-adter-write 一致性模型
6.1 准备工作
调整docker-compose-minio.yaml文件，内容如下：
version: '3.7'
# Settings and configurations that are common for all containers
x-minio-common: &minio-common
image: quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z
command: server --console-address ":9090" http://minio{1...4}/data{1...4}
expose:
- "9000"
- "9090"
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
TZ: Asia/Shanghai
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
networks:
- pk_net
# 启动4个minio服务器实例的docker容器
# 使用nginx反向代理，负载均衡，通过端口9000可以访问
# it through port 9000.
services:
minio1:
<<: *minio-common
hostname: minio1
container_name: minio1
volumes:
- /data1/data1:/data1
- /data1/data2:/data2
- /data1/data3:/data3
- /data1/data4:/data4
minio2:
<<: *minio-common
hostname: minio2
container_name: minio2
volumes:
- /data2/data1:/data1
- /data2/data2:/data2
- /data2/data3:/data3
- /data2/data4:/data4
minio3:
<<: *minio-common
hostname: minio3
container_name: minio3
volumes:
- /data3/data1:/data1
- /data3/data2:/data2
- /data3/data3:/data3
- /data3/data4:/data4
minio4:
<<: *minio-common
hostname: minio4
container_name: minio4
volumes:
- /data4/data1:/data1
- /data4/data2:/data2
- /data4/data3:/data3
- /data4/data4:/data4
nginx:
image: nginx:1.19.2-alpine
hostname: nginx
container_name: nginx
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
ports:
- "9000:9000"
- "9090:9090"
depends_on:
- minio1
- minio2
- minio3
- minio4
networks:
pk_net:
external: true
参数说明：x-minio-common表示所有容器的设置和配置 启动4个minio服务器实例的docker容器 使用nginx反向代理，负载均衡，通过端口9000可以访问
nginx.conf内容如下：
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 4096;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
# include /etc/nginx/conf.d/*.conf;
upstream minio {
server minio1:9000;
server minio2:9000;
server minio3:9000;
server minio4:9000;
}
upstream console {
ip_hash;
server minio1:9090;
server minio2:9090;
server minio3:9090;
server minio4:9090;
}
server {
listen 9000;
listen [::]:9000;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://minio;
}
}
server {
listen 9090;
listen [::]:9090;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-NginX-Proxy true;
# This is necessary to pass the correct IP to be hashed
real_ip_header X-Real-IP;
proxy_connect_timeout 300;
# To support websocket
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
chunked_transfer_encoding off;
proxy_pass http://console;
}
}
}
6.2 部署
$ docker-compose -f docker-compose-minio.yaml up -d
Creating minio4 ... done
Creating minio2 ... done
Creating minio3 ... done
Creating minio1 ... done
Creating test_nginx_1 ... done
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a6bda0af2d24 nginx:1.19.2-alpine "/docker-entrypoint.…" 7 seconds ago Up 6 seconds 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp, 80/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp test_nginx_1
239c4bdeb151 quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z "/usr/bin/docker-ent…" 9 seconds ago Up 6 seconds (health: starting) 9000/tcp, 9090/tcp minio1
0c5050be7fd3 quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z "/usr/bin/docker-ent…" 9 seconds ago Up 6 seconds (health: starting) 9000/tcp, 9090/tcp minio2
e65601ce11c0 quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z "/usr/bin/docker-ent…" 9 seconds ago Up 7 seconds (health: starting) 9000/tcp, 9090/tcp minio3
f75733c2da78 quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z "/usr/bin/docker-ent…" 9 seconds ago Up 7 seconds (health: starting) 9000/tcp, 9090/tcp minio4
七、MinIO分布式部署(多节点多驱动)--生产推荐
在单机上部署只能保证磁盘损坏的情况下，文件不丢失，并不能解决单点故障的问题，所以我们下面为了避免单点故障导致服务不可用，把minio服务改成真正分布式部署。MinIO强烈推荐生产集群由 minimum 4 个组成 minio server 节点在服务器池。可以采用docker-compose部署，也可采用二进制部署，最后使用nginx进行负载均衡就行了。这里我们还是采用docker-compose为例。
7.1 准备工作
服务器1-4如下：
docker-compose-minio.yaml文件，内容如下：
version: '3.7'
services:
minio:
image: quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z
container_name: minio
hostname: minio
restart: always
command: server --console-address ":9090" http://192.168.0.1:9000/data{1...4}
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
TZ: Asia/Shanghai
volumes:
- /minio1/data1:/data1
- /minio2/data2:/data2
- /minio3/data3:/data3
- /minio4/data4:/data4
- /data/minio/config:/root/.minio
ports:
- "9000:9000"
- "9090:9090"
networks:
- pk_net
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
networks:
pk_net:
external: true
服务器1-4都采用以上docker-compose文件部署。/minio1/data1、/minio2/data2、/minio3/data2、/minio4/data2表示分别挂载的四块磁盘
docker-compose-nginx.yaml文件，内容如下：
version: '3.7'
services:
nginx:
image: nginx:1.19.2-alpine
hostname: nginx
container_name: nginx
restart: always
environment:
TZ: Asia/Shanghai
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
ports:
- "9000:9000"
- "9090:9090"
networks:
- pk_net
networks:
pk_net:
external: true
nginx.conf内容如下：
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 4096;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
# include /etc/nginx/conf.d/*.conf;
upstream minio {
server 192.168.0.1:9000;
server 192.168.0.2:9000;
server 192.168.0.3:9000;
server 192.168.0.4:9000;
}
upstream console {
ip_hash;
server 192.168.0.1:9090;
server 192.168.0.2:9090;
server 192.168.0.3:9090;
server 192.168.0.4:9090;
}
server {
listen 9000;
listen [::]:9000;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://minio;
}
}
server {
listen 9090;
listen [::]:9090;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-NginX-Proxy true;
# This is necessary to pass the correct IP to be hashed
real_ip_header X-Real-IP;
proxy_connect_timeout 300;
# To support websocket
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
chunked_transfer_encoding off;
proxy_pass http://console;
}
}
}
7.2 部署
分别部署四台minio服务
$ docker-compose -f docker-compose-minio.yaml up -d
部署nginx服务
$ docker-compose -f docker-compose-nginx.yaml up -d
八、优化
nginx也存在单节点故障，可以进行高可用配置。
非云上：
1.使用 Keepalived 和 HAproxy 负载minio
2.使用 Keepalived 和 Nginx 负载minio
云上：
1.使用 Keepalived 和 HAproxy 负载minio
2.使用 Keepalived 和 Nginx 负载minio
3.可以直接使用云上的lb，比如阿里云slb，腾讯云elb，青云lb等负载minio（推荐）
ps: 有些云可能不支持keepalived
