linux运维、架构之路-分布式存储GlusterFS
一、GlusterFS介绍
GlusterFS是一种全对称的开源分布式文件系统，所谓全对称是指GlusterFS采用弹性哈希算法，没有中心节点，所有节点全部平等。GlusterFS配置方便，稳定性好，可轻松达到PB级容量以及数千个节点。PB级容量、高可用、读写性能、基于文件系统级别共享、分布式、无metadata(元数据)的存储方式。
二、GlusterFS特性
全对称架构、支持多种卷类型(类似RAID0/1/5/10/01)、支持卷级别的压缩、支持FUSE、支持NFS、支持SMB、支持Hadoop、支持OpenStack。
三、GlusterFS概念说明
- birck：GlusterFS的基本元素，以节点服务器目录形式展现；
- volume：多个brick的逻辑集合；
- metadata：元数据，用于描述文件、目录等的信息；
- self-heal：用于后台运行检测副本卷中文件和目录的不一致性并解决这些不一致；
- FUSE：Filesystem Userspace是一个可加载的内核模块，其支持非特权用户创建自己的文件系统而不需要修改内核代码通过在用户空间运行文件系统的代码通过FUSE代码与内核进行桥接；
- Gluster Server：数据存储服务器，即组成GlusterFS存储集群的节点；
- Gluster Client：使用GlusterFS存储服务的服务器，如KVM、OpenStack、LB RealServer、HA node。
四、GlusterFS部署
1、服务器规划
|
系统 |
IP地址 |
主机名 |
CPU |
内存 |
|
CentOS 7.5 |
192.168.56.10 |
server1 |
2C |
2G |
|
CentOS 7.5 |
192.168.56.11 |
server2 |
2C |
2G
|
①/etc/hosts解析
cat >> /etc/hosts <<EOF 192.168.56.10 server1 192.168.56.11 server2 EOF
②关闭防火墙、SElinux等
setenforce 0 systemctl stop firewalld systemctl disable firewalld sed -i 's/enforcing/disabled/g' /etc/sysconfig/selinux
2、GlusterFS安装部署
yum -y install epel-release yum -y install centos-release-gluster yum -y install glusterfs glusterfs-server glusterfs-fuse
①客户端安装
yum -y install glusterfs glusterfs-fuse
②启动服务(所有节点)
systemctl enable glusterd.service && systemctl start glusterd.service
③GlusterFS节点信任
假设现在只有两个节点
server1上执行：gluster peer probe server2
server2上执行：gluster peer probe server1
从集群中移除节点
gluster peer detach server2 #从集群中去除节点(该节点中不能存在卷中正在使用的brick)
④创建存储卷并启动(任意一个节点执行)
mkdir /data/gluster_volumes -p #准备目录创建存储卷 gluster volume create gluster_volumes replica 2 server1:/data/gluster_volumes server2:/data/gluster_volumes #创建存储卷 gluster volume start gluster_volumes #启动卷
注：GlusterFS支持多种存储类型，不同的类型存储数据的方式是不同的，我这里使用的是Replicate，即两台节点机器存储内容完全一致。这样做的好处是，如果出现单机故障，那么另一台节点上也有完整数据。原理图如下：
多个文件在多个brick上复制多份，brick的数目要与需要复制的份数相等，replica = brick，建议brick分布在不同的服务器上。应用场景：对可靠性高和读写性能要求高的场景
⑤查看存储卷信息
[root@server1 ~]# gluster volume info Volume Name: gluster_volumes Type: Replicate Volume ID: 80c40e90-d912-414a-a510-2f30cb746c03 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: server1:/data/gluster_volumes Brick2: server2:/data/gluster_volumes Options Reconfigured: transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: off
⑥查看集群状态
[root@server1 ~]# gluster peer status Number of Peers: 1 Hostname: server2 Uuid: 0eda1c59-cad7-4e33-a028-99027c6e99b4 State: Peer in Cluster (Connected)
3、配置客户端使用卷
①将卷挂载到本地目录
mkdir /app && mount -t glusterfs server1:/gluster_volumes /app [root@server1 ~]# df -h 文件系统 容量 已用 可用 已用% 挂载点 /dev/sda3 17G 2.0G 16G 12% / devtmpfs 981M 0 981M 0% /dev tmpfs 992M 0 992M 0% /dev/shm tmpfs 992M 9.6M 982M 1% /run tmpfs 992M 0 992M 0% /sys/fs/cgroup /dev/sda1 1014M 124M 891M 13% /boot tmpfs 199M 0 199M 0% /run/user/0 server1:/gluster_volumes 17G 2.1G 15G 13% /app
②设置开机自动挂载
vim /etc/fstab加入： server1:/gluster_volumes /app glusterfs defaults,_netdev 0 0
③对卷目录写入文件测试
for i in `seq -w 1 100`; do cp -rp /var/log/messages /app/copy-test-$i; done
④查看每个节点的GlusterFS卷目录
[root@server1 ~]# ls /data/gluster_volumes copy-test-001 copy-test-014 copy-test-027 copy-test-040 copy-test-053 copy-test-066 copy-test-079 copy-test-092 copy-test-002 copy-test-015 copy-test-028 copy-test-041 copy-test-054 copy-test-067 copy-test-080 copy-test-093 copy-test-003 copy-test-016 copy-test-029 copy-test-042 copy-test-055 copy-test-068 copy-test-081 copy-test-094 copy-test-004 copy-test-017 copy-test-030 copy-test-043 copy-test-056 copy-test-069 copy-test-082 copy-test-095 copy-test-005 copy-test-018 copy-test-031 copy-test-044 copy-test-057 copy-test-070 copy-test-083 copy-test-096 copy-test-006 copy-test-019 copy-test-032 copy-test-045 copy-test-058 copy-test-071 copy-test-084 copy-test-097 copy-test-007 copy-test-020 copy-test-033 copy-test-046 copy-test-059 copy-test-072 copy-test-085 copy-test-098 copy-test-008 copy-test-021 copy-test-034 copy-test-047 copy-test-060 copy-test-073 copy-test-086 copy-test-099 copy-test-009 copy-test-022 copy-test-035 copy-test-048 copy-test-061 copy-test-074 copy-test-087 copy-test-100 copy-test-010 copy-test-023 copy-test-036 copy-test-049 copy-test-062 copy-test-075 copy-test-088 copy-test-011 copy-test-024 copy-test-037 copy-test-050 copy-test-063 copy-test-076 copy-test-089 copy-test-012 copy-test-025 copy-test-038 copy-test-051 copy-test-064 copy-test-077 copy-test-090 copy-test-013 copy-test-026 copy-test-039 copy-test-052 copy-test-065 copy-test-078 copy-test-091
[root@server2 ~]# ls /data/gluster_volumes/ copy-test-001 copy-test-014 copy-test-027 copy-test-040 copy-test-053 copy-test-066 copy-test-079 copy-test-092 copy-test-002 copy-test-015 copy-test-028 copy-test-041 copy-test-054 copy-test-067 copy-test-080 copy-test-093 copy-test-003 copy-test-016 copy-test-029 copy-test-042 copy-test-055 copy-test-068 copy-test-081 copy-test-094 copy-test-004 copy-test-017 copy-test-030 copy-test-043 copy-test-056 copy-test-069 copy-test-082 copy-test-095 copy-test-005 copy-test-018 copy-test-031 copy-test-044 copy-test-057 copy-test-070 copy-test-083 copy-test-096 copy-test-006 copy-test-019 copy-test-032 copy-test-045 copy-test-058 copy-test-071 copy-test-084 copy-test-097 copy-test-007 copy-test-020 copy-test-033 copy-test-046 copy-test-059 copy-test-072 copy-test-085 copy-test-098 copy-test-008 copy-test-021 copy-test-034 copy-test-047 copy-test-060 copy-test-073 copy-test-086 copy-test-099 copy-test-009 copy-test-022 copy-test-035 copy-test-048 copy-test-061 copy-test-074 copy-test-087 copy-test-100 copy-test-010 copy-test-023 copy-test-036 copy-test-049 copy-test-062 copy-test-075 copy-test-088 copy-test-011 copy-test-024 copy-test-037 copy-test-050 copy-test-063 copy-test-076 copy-test-089 copy-test-012 copy-test-025 copy-test-038 copy-test-051 copy-test-064 copy-test-077 copy-test-090 copy-test-013 copy-test-026 copy-test-039 copy-test-052 copy-test-065 copy-test-078 copy-test-091
如果在两个节点上都看到有101个文件，到此GlusterFS部署结束。
五、配置GlusterFS作为k8s的持久化存储
1、创建endpoint
[root@k8s-node1 gluster]# cat glusterfs-ep.yaml apiVersion: v1 kind: Endpoints metadata: name: glusterfs namespace: default subsets: - addresses: - ip: 192.168.56.175 - ip: 192.168.56.176 ports: - port: 49152 protocol: TCP
2、创建service
[root@k8s-node1 gluster]# cat glusterfs-svc.yaml apiVersion: v1 kind: Service metadata: name: glusterfs namespace: default spec: ports: - port: 49152 protocol: TCP targetPort: 49152 sessionAffinity: None type: ClusterIP
3、创建PV
[root@k8s-node1 gluster]# cat glusterfs-pv.yaml apiVersion: v1 kind: PersistentVolume metadata: name: gluster labels: type: glusterfs spec: capacity: storage: 100Gi accessModes: - ReadWriteMany glusterfs: endpoints: "glusterfs" path: "gluster_volumes" readOnly: false
4、创建PVC
[root@k8s-node1 gluster]# cat glusterfs-pvc.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: gluster spec: accessModes: - ReadWriteMany resources: requests: storage: 20Gi
5、创建应用挂载测试
apiVersion: v1 kind: Pod metadata: name: nginx-demo spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 volumeMounts: - name: nginx-volume mountPath: /usr/share/nginx/html volumes: - name: nginx-volume persistentVolumeClaim: claimName: gluster
成功最有效的方法就是向有经验的人学习！
