k8s离线部署minio集群

1.minio集群介绍

MinIO集群是一个高性能、分布式的对象存储系统，采用去中心化无共享架构，所有节点对等且数据均匀分布，避免单点故障。它基于纠删码技术，将数据分片并冗余存储，允许在部分节点故障时（如半数以下磁盘损坏）仍能恢复数据，确保高可用性和容错性。集群支持水平扩展，可通过添加节点轻松扩容，兼容Amazon S3 API，适用于云原生和大规模非结构化数据存储场景。此外，MinIO集群部署灵活，最少需4个节点，并提供统一命名空间，简化数据管理。

2.部署环境

IP	节点	操作系统	k8s版本	minio版本	docker版本
172.16.4.85	master1	centos7.8	1.23.17		20.10.9
172.16.4.86	node1	centos7.8	1.23.17	minio:v20220104	20.10.9
172.16.4.87	node2	centos7.8	1.23.17	minio:v20220104	20.10.9
172.16.4.89	node3	centos7.8	1.23.17	minio:v20220104	20.10.9
172.16.4.90	node4	centos7.8	1.23.17	minio:v20220104	20.10.9

3.minio集群部署

3.1 nfs部署

centos7安装nfs

yum install -y nfs-utils

mkdir -p /nfs_share/k8s/minio/pv{1..4}

nfs配置文件编辑

[root@localhost minio]# cat /etc/exports
/nfs_share/k8s/minio/pv1 *(rw,sync,no_subtree_check,no_root_squash)
/nfs_share/k8s/minio/pv2 *(rw,sync,no_subtree_check,no_root_squash)
/nfs_share/k8s/minio/pv3 *(rw,sync,no_subtree_check,no_root_squash)

启动nfs服务

# 启动 NFS 服务
systemctl start nfs-server
# 设置 NFS 服务在系统启动时自动启动
systemctl enable nfs-server

加载配置文件，并输出

[root@localhost minio]# exportfs -r
[root@localhost minio]# exportfs -v
/nfs_share/k8s/minio/pv1
		<world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/nfs_share/k8s/minio/pv2
		<world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/nfs_share/k8s/minio/pv3
		<world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/nfs_share/k8s/minio/pv4
		<world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)

3.2 创建namespace

kubectl create namespace minio

3.3 minio部署pv

apiVersion: v1
kind: PersistentVolume
metadata:
  name: minio-nfs-pv-1
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain  # 保留数据，避免误删
  storageClassName: minio-nfs-storage    # 必须与 StatefulSet 中的 storageClassName 一致
  nfs:
    path: /nfs_share/k8s/minio/pv1      # 确保 NFS 路径存在且可访问
    server: 172.16.4.60
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: minio-nfs-pv-2
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: minio-nfs-storage
  nfs:
    path: /nfs_share/k8s/minio/pv2
    server: 172.16.4.60
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: minio-nfs-pv-3
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: minio-nfs-storage
  nfs:
    path: /nfs_share/k8s/minio/pv3
    server: 172.16.4.60
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: minio-nfs-pv-4 
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany 
  storageClassName: minio-nfs-storage
  nfs:
    path: /nfs_share/k8s/minio/pv4
    server: 172.16.4.60

kubectl apply -f minio-pv.yaml

3.4 minio部署headless service

# minio-headless-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: minio-headless
  namespace: minio
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  ports:
    - name: api
      port: 29001
    - name: console
      port: 29000
  selector:
    app: minio

kubectl apply -f minio-headless.yaml

3.5 minio部署statefulset

# minio-ss.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: minio
  namespace: minio
spec:
  serviceName: minio-headless
  replicas: 4
  podManagementPolicy: Parallel
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      labels:
        app: minio
    spec:
      initContainers:
      - name: wait-for-dns
        image: 172.16.4.17:8090/ltzx/busybox:latest
        command: ['sh', '-c', 'until nslookup minio-headless.minio.svc.cluster.local; do echo "等待 DNS 就绪..."; sleep 2; done']
      containers:
      - name: minio
        image: 172.16.4.17:8090/public/minio:v20220104
        command:
        - /bin/sh
        - -c
        - "minio server --address :29001 --console-address :29000 http://minio-{0..3}.minio-headless.minio.svc.cluster.local/data"
        env:
        - name: MINIO_ROOT_USER
          value: "admin"
        - name: MINIO_ROOT_PASSWORD
          value: "keshen@1234"
        ports:
        - containerPort: 29001
          name: api
        - containerPort: 29000
          name: console
        startupProbe:
          exec:
            command:
            - sh
            - -c
            - "test -d /data/.minio.sys"
          failureThreshold: 30
          periodSeconds: 10
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteMany" ]
      storageClassName: minio-nfs-storage
      resources:
        requests:
          storage: 10Gi

kubectl apply -f minio-ss.yaml

3.6 minio部署外部service（业务连接，具有自动负载均衡能力）

apiVersion: v1
kind: Service
metadata:
  name: minio-svc
  namespace: minio
  labels:
    app: minio
spec:
  type: NodePort
  ports:
    - name: api
      port: 29001
      targetPort: 29001
      nodePort: 30001
      protocol: TCP
    - name: console
      port: 29000
      targetPort: 29000
      nodePort: 30002
      protocol: TCP
  selector:
    app: minio

kubectl apply -f minio-svc.yaml

3.7 minio状态查看

[root@master1 minio-n6]# kubectl get pv | grep minio
minio-nfs-pv-1          10Gi       RWX            Retain           Bound    minio/data-minio-0                       minio-nfs-storage            3h37m
minio-nfs-pv-2          10Gi       RWX            Retain           Bound    minio/data-minio-1                       minio-nfs-storage            3h37m
minio-nfs-pv-3          10Gi       RWX            Retain           Bound    minio/data-minio-2                       minio-nfs-storage            3h37m
minio-nfs-pv-4          10Gi       RWX            Retain           Bound    minio/data-minio-3                       minio-nfs-storage            3h37m
[root@master1 minio-n6]# kubectl get pvc -n minio
NAME           STATUS   VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS        AGE
data-minio-0   Bound    minio-nfs-pv-1   10Gi       RWX            minio-nfs-storage   3h37m
data-minio-1   Bound    minio-nfs-pv-2   10Gi       RWX            minio-nfs-storage   3h37m
data-minio-2   Bound    minio-nfs-pv-3   10Gi       RWX            minio-nfs-storage   3h37m
data-minio-3   Bound    minio-nfs-pv-4   10Gi       RWX            minio-nfs-storage   3h37m
[root@master1 minio-n6]# kubectl get svc -n minio
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                           AGE
minio-headless   ClusterIP   None             <none>        29001/TCP,29000/TCP               8h
minio-svc        NodePort    10.106.118.183   <none>        29001:30001/TCP,29000:30002/TCP   3h22m
[root@master1 minio-n6]# kubectl get sts -n minio
NAME    READY   AGE
minio   4/4     3h36m
[root@master1 minio-n6]# kubectl get pods -n minio -o wide 
NAME      READY   STATUS    RESTARTS   AGE     IP               NODE    NOMINATED NODE   READINESS GATES
minio-0   1/1     Running   0          3h36m   10.244.3.93      node4   <none>           <none>
minio-1   1/1     Running   0          3h36m   10.244.135.62    node3   <none>           <none>
minio-2   1/1     Running   0          3h36m   10.244.104.59    node2   <none>           <none>
minio-3   1/1     Running   0          3h36m   10.244.166.184   node1   <none>           <none>

3.8 minio集群验证

通过minio pod的日志查看容器状态，看到下边状态就说明没问题

#minio-0 pod
[root@master1 minio-n6]# kubectl logs -f minio-0 -n minio

 You are running an older version of MinIO released 3 years ago 
 Update: 
Run `mc admin update` 


Formatting 1st pool, 1 set(s), 4 drives per set.
Waiting for all MinIO sub-systems to be initialized.. lock acquired
Automatically configured API requests per node based on available memory on the system: 245
All MinIO sub-systems initialized successfully
Waiting for all MinIO IAM sub-system to be initialized.. lock acquired
Status:         4 Online, 0 Offline. 
API: http://10.244.3.93:29001  http://127.0.0.1:29001 

Console: http://10.244.3.93:29000 http://127.0.0.1:29000 

Documentation: https://docs.min.io
#minio-1 pod
[root@master1 minio-n6]# kubectl logs -f minio-1 -n minio
Waiting for the first server to format the disks.

 You are running an older version of MinIO released 3 years ago 
 Update: 
Run `mc admin update` 


Waiting for all MinIO sub-systems to be initialized.. lock acquired
Automatically configured API requests per node based on available memory on the system: 192
All MinIO sub-systems initialized successfully
Status:         4 Online, 0 Offline. 
API: http://10.244.135.62:29001  http://127.0.0.1:29001 

Console: http://10.244.135.62:29000 http://127.0.0.1:29000 

Documentation: https://docs.min.io
Waiting for all MinIO IAM sub-system to be initialized.. lock acquired
#minio-2 pod
[root@master1 minio-n6]# kubectl logs -f minio-2 -n minio

 You are running an older version of MinIO released 3 years ago 
 Update: 
Run `mc admin update` 


Waiting for all MinIO sub-systems to be initialized.. lock acquired
Automatically configured API requests per node based on available memory on the system: 196
All MinIO sub-systems initialized successfully
Waiting for all MinIO IAM sub-system to be initialized.. lock acquired
Status:         4 Online, 0 Offline. 
API: http://10.244.104.59:29001  http://127.0.0.1:29001 

Console: http://10.244.104.59:29000 http://127.0.0.1:29000 

Documentation: https://docs.min.io
#minio-3 pod
[root@master1 minio-n6]# kubectl logs -f minio-3 -n minio
Waiting for the first server to format the disks.

 You are running an older version of MinIO released 3 years ago 
 Update: 
Run `mc admin update` 


Waiting for all MinIO sub-systems to be initialized.. lock acquired
Automatically configured API requests per node based on available memory on the system: 193
All MinIO sub-systems initialized successfully
Waiting for all MinIO IAM sub-system to be initialized.. lock acquired
Status:         4 Online, 0 Offline. 
API: http://10.244.166.184:29001  http://127.0.0.1:29001 

Console: http://10.244.166.184:29000 http://127.0.0.1:29000 

Documentation: https://docs.min.io

通过web 控制台验证，通过下边地址，都可以打开minio控制台，另外在控制台可以看到有4个service

http://172.16.4.85:30002
http://172.16.4.86:30002
http://172.16.4.87:30002
http://172.16.4.89:30002
http://172.16.4.90:30002

4.minio集群地址

内部pod地址，通过 Headless Service.namespace 进行域名解析

minio-0.minio-headless.minio.svc.cluster.local
minio-1.minio-headless.minio.svc.cluster.local
minio-2.minio-headless.minio.svc.cluster.local
minio-3.minio-headless.minio.svc.cluster.local

集群内部地址（Headless Service）

minio-headless.minio.svc.cluster.local

业务连接集群地址（使用service地址，因为cluster IP，k8s自动负载均衡，依赖kube-proxy）

minio-svc.minio.svc.cluster.local

外部nodeport连接

http://172.16.4.85:30002
http://172.16.4.86:30002
http://172.16.4.87:30002
http://172.16.4.89:30002
http://172.16.4.90:30002

至此minio集群就部署完成了！！！

posted @ 2025-02-28 17:52 Leonardo-li 阅读(12) 评论(0) 编辑收藏举报

刷新页面返回顶部

登录后才能查看或发表评论，立即登录或者逛逛博客园首页

相关博文：

· k8s部署redis cluster集群

· k8s部署及周边服务（详细总结篇）

· 如何通过minio operator在k8s中部署minio租户（tenant）集群

· k8s 1.28.2 集群部署 MinIO 分布式集群

· 21.ubuntu16.04 Minio 集群搭建

阅读排行：
· winform 绘制太阳，地球，月球运作规律
· TypeScript + Deepseek 打造卜卦网站：技术与玄学的结合
· AI 智能体引爆开源社区「GitHub 热点速览」
· 写一个简单的SQL生成工具
· Manus的开源复刻OpenManus初探

历史上的今天：
2018-02-28 使用 iftop 命令监控网络带宽
2018-02-28 用docker容器来制作nginx镜像

公告

昵称： Leonardo-li
园龄： 7年4个月
粉丝： 34
关注： 15

+加关注

2025年3月

日

一

二

三

四

五

六

Leonardo-li

k8s离线部署minio集群

1.minio集群介绍

2.部署环境

3.minio集群部署

3.1 nfs部署

3.2 创建namespace

3.3 minio部署pv

3.4 minio部署headless service

3.5 minio部署statefulset

3.6 minio部署外部service（业务连接，具有自动负载均衡能力）

3.7 minio状态查看

3.8 minio集群验证

4.minio集群地址

公告

搜索

常用链接

我的标签

随笔分类

随笔档案

阅读排行榜

评论排行榜

推荐排行榜

最新评论