GlusterFS+Keepalived实现存储高可用
l 环境准备
1.服务器列表信息
|
Ip |
Hostname |
存储 |
系统 |
Vip |
|
192.168.1.42 |
data-node-01 |
/dev/sdb1 |
Centos7 |
192.168.1.99 说明：用于对外提供存储服务 |
|
192.168.1.49 |
data-node-02 |
/dev/sdb1 |
Centos7 |
|
|
192.168.1.51 |
web-node-12 |
客户端 |
Centos7 |
192.168.1.42 data-node-01
192.168.1.49 data-node-02
建立GlusterFS存储挂载点（data-node-01和data-node-02都要做此操作）
mkdir -p /glusterfs/storage1
echo "/dev/sdb1 /glusterfs/storage1 xfs defaults 0 0" >> /etc/fstab
mount -a
l 安装GlusterFS服务端软件
3.安装gluster源，并安装glusterfs及相关软件包（data-node-01和data-node-02都要做此操作）
yum install centos-release-gluster -y
yum install glusterfs glusterfs-server glusterfs-cli glusterfs-geo-replication glusterfs-rdma -y
4.启动Glusterd服务（data-node-01和data-node-02都要做此操作）
systemctl start glusterd
5.在任意一个节点上添加信任节点（这里我们选择在data-node-01上添加，注意添加节点时不用添加当前节点，只需要添加其他节点就行）
gluster peer probe data-node-02 //例如：本实验只在data-node-01上添加data-node-02
gluster peer status
6.在任意一个节点上建立复制卷（这里我们选择data-node-01）
[root@localhost ~]# mkdir /glusterfs/storage1/rep_vol1 //分别在data-node-01和data-node-02上创建目录
gluster volume create rep_vol1 replica 2 data-node-01:/glusterfs/storage1/rep_vol1 data-node-02:/glusterfs/storage1/rep_vol1
7.启动复制卷
gluster volume start rep_vol1
8.查看复制卷状态
gluster volume status
gluster volume info
l 安装GlusterFS客户端软件（web-node-12上的操作）
9.客户端安装GlusterFS客户端软件
yum install glusterfs-fuse
10.客户端测试复制卷数据存储
for i in `seq -w 1 3`;do cp -rp /var/log/messages /data/test-$i;done
[root@localhost ~]# ls /data/
111 1.txt 2.txt anaconda-ks.cfg test-1 test-2 test-3
l 安装与配置Keepalived
11.安装Keepalived
yum -y install keepalived
12.启动keepalived服务
systemctl start keepalived
13.主节点keepalived配置
! Configuration File for keepalived
global_defs {
notification_email {
mail@huangming.org
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id GFS_HA_MASTER
vrrp_skip_check_adv_addr
}
vrrp_sync_group GFS_HA_GROUP {
group {
GFS_HA_1
}
}
vrrp_script monitor_glusterfs_status {
script "/etc/keepalived/scripts/monitor_glusterfs_status.sh"
interval 5
fall 3
rise 1
weight 20
}
vrrp_instance GFS_HA_1 {
state BACKUP
interface ens34
virtual_router_id 107
priority 100
advert_int 2
nopreempt
authentication {
auth_type PASS
auth_pass 11112222
}
virtual_ipaddress {
192.168.1.99/24 dev ens34
}
track_script {
monitor_glusterfs_status
}
track_interface {
ens34
}
notify_master "/etc/keepalived/scripts/keepalived_notify.sh master"
notify_backup "/etc/keepalived/scripts/keepalived_notify.sh backup"
notify_fault "/etc/keepalived/scripts/keepalived_notify.sh fault"
notify_stop "/etc/keepalived/scripts/keepalived_notify.sh stop"
}
14.备节点keepalived配置
! Configuration File for keepalived
global_defs {
notification_email {
mail@huangming.org
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id GFS_HA_MASTER
vrrp_skip_check_adv_addr
}
vrrp_sync_group GFS_HA_GROUP {
group {
GFS_HA_1
}
}
vrrp_script monitor_glusterfs_status {
script "/etc/keepalived/scripts/monitor_glusterfs_status.sh"
interval 5
fall 3
rise 1
weight 20
}
vrrp_instance GFS_HA_1 {
state BACKUP
interface ens34
virtual_router_id 107
priority 90
advert_int 2
authentication {
auth_type PASS
auth_pass 11112222
}
virtual_ipaddress {
192.168.1.99/24 dev ens34
}
track_script {
monitor_glusterfs_status
}
track_interface {
ens34
}
notify_master "/etc/keepalived/scripts/keepalived_notify.sh master"
notify_backup "/etc/keepalived/scripts/keepalived_notify.sh backup"
notify_fault "/etc/keepalived/scripts/keepalived_notify.sh fault"
notify_stop "/etc/keepalived/scripts/keepalived_notify.sh stop"
}
15.keepalived vrrp监控脚本
cat /etc/keepalived/scripts/monitor_glusterfs_status.sh
#!/bin/bash
#check glusterfsd and glusterd process
systemctl status glusterd &>/dev/null
if [ $? -eq 0 ];then
systemctl status glusterfsd &>/dev/null
if [ $? -eq 0 ];then
exit 0
else
exit 2
fi
else
systemctl start glusterd &>/dev/null
systemctl stop keepalived &>/dev/null && exit 1
fi
16.keepalived通知脚本(管理Glusterd服务)
cat /etc/keepalived/scripts/keepalived_notify.sh
#!/bin/bash
#keepalived script for glusterd
master() {
systemctl status glusterd
if [ $? -ne 0 ];then
systemctl start glusterd
else
systemctl restart glusterd
fi
}
backup() {
systemctl status glusterd
if [ $? -ne 0 ];then
systemctl start glusterd
fi
}
case $1 in
master)
master
;;
backup)
backup
;;
fault)
backup
;;
stop)
backup
systemctl restart keepalived
;;
*)
echo $"Usage: $0 {master|backup|fault|stop}"
esac
l 测试Keepalived自动接管GlusterFS服务及存储的可用性
17.重新启动keepalived服务
systemctl restart keepalived.service
18.查看VIP接管情况
## 节点1上
[root@data-node-01 ~]# ip a show dev ens34
3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:b2:b5:2a brd ff:ff:ff:ff:ff:ff
inet 192.168.1.42/24 brd 192.168.1.255 cope global ens34
valid_lft forever preferred_lft forever
inet 192.168.1.99/24 scope global secondary ens34
valid_lft forever preferred_lft forever
inet6 fe80::ce9a:ee2e:7b6c:a6bb/64 scope link
valid_lft forever preferred_lft forever
## 节点2上
[root@data-node-02 ~]# ip a show dev ens34
3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:ba:42:cf brd ff:ff:ff:ff:ff:ff
inet192.168.1.49/24 brd 192.168.1.255 scope global ens34
valid_lft forever preferred_lft forever
inet6 fe80::e23:ce0:65c3:ffbf/64 scope link
valid_lft forever preferred_lft forever
19.客户端上使用VIP挂载GlusterFS提供的复制卷，并测试是否可用
mount -t glusterfs 192.168.1.99:rep_vol1 /data/
[root@localhost ~]# ls /data/
111 1.txt 2.txt anaconda-ks.cfg test test-1 test-2 test-3
[root@localhost ~]# mkdir /data/test
[root@localhost ~]# echo 1111 >/data/test/1.txt
[root@localhost ~]# ls /data/test
1.txt
[root@localhost ~]# cat /data/test/1.txt
1111
20.查看GluserFS节点复制卷的使用情况
[root@data-node-02 ~]# ls /glusterfs/storage1/rep_vol1/
111 1.txt 2.txt anaconda-ks.cfg test test-1 test-2 test-3
21.测试GlusterFS服务故障转移, 将主节点(节点1)关机或重启，查看GlusterFS服务与VIP是否转移至节点2
root@data-node-01 ~]# reboot
[root@data-node-02 ~]# ip a show dev ens34
3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:ba:42:cf brd ff:ff:ff:ff:ff:ff
inet 192.168.1.49/24 brd 192.168.1.255 scope global ens34
valid_lft forever preferred_lft forever
inet 192.168.1.99/24 cope global secondary ens34
valid_lft forever preferred_lft forever
inet6 fe80::e23:ce0:65c3:ffbf/64 scope link
valid_lft forever preferred_lft forever
22.在客户端上测试存储是否仍然可用
[root@localhost ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/cl-root xfs 40G 1.2G 39G 3% /
devtmpfs devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs tmpfs 1.9G 8.6M 1.9G 1% /run
tmpfs tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda1 xfs 1014M 139M 876M 14% /boot
tmpfs tmpfs 378M 0 378M 0% /run/user/0
192.168.1.99:rep_vol1 fuse.glusterfs 10G 136M 9.9G 2% /data
[root@localhost ~]# ls /data/
111 1.txt 2.txt anaconda-ks.cfg test test-1 test-2 test-3
[root@localhost ~]# touch /data/test.log
[root@localhost ~]# ls -l /data/
total 964
drwxr-xr-x 3 root root 4096 Aug 27 21:58 111
-rw-r--r-- 1 root root 10 Aug 27 21:23 1.txt
-rw-r--r-- 1 root root 6 Aug 27 21:36 2.txt
-rw------- 1 root root 2135 Aug 27 21:44 anaconda-ks.cfg
drwxr-xr-x 2 root root 4096 Aug 27 22:59 test
-rw------- 1 root root 324951 Aug 27 21:23 test-1
-rw------- 1 root root 324951 Aug 27 21:23 test-2
-rw------- 1 root root 324951 Aug 27 21:23 test-3
-rw-r--r-- 1 root root 0 Aug 27 23:05 test.log
23.查看节点1状态
[root@data-node-01 ~]# ip a show dev ens34
3: ens34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:b2:b5:2a brd ff:ff:ff:ff:ff:ff
inet 192.168.1.42/24 brd 192.168.1.255 scope global ens34
valid_lft forever preferred_lft forever
inet6 fe80::ce9a:ee2e:7b6c:a6bb/64 scope link
valid_lft forever preferred_lft forever
24.重新启动keepalived服务
[root@data-node-01 ~]# systemctl start keepalived.service
说明总结：
由此可见当节点1故障恢复后，节点1的keepalived会进入到备用状态，同时继续监管GlusterFS服务，当节点2故障时会将服务、存储和VIP切换到节点1，继续对外提供存储服务，从而实现存储的高可用
