nfs+DRBD+corosync+pacemaker 实现高可用（ha）的nfs集群
https://blog.51cto.com/legehappy/1976565
nfs+DRBD+corosync+pacemaker 实现高可用（ha）的nfs集群（centos7）摘要：
环境介绍
借助pcs安装与配置corosync和pacemaker（pcs只是一个管理工具）
DRBD安装配置参考之前的博客《DRBD-MYSQL分布式块设备实现高可用》
http://legehappy.blog.51cto.com/13251607/1975804
Nfs的安装和配置
Crmsh安装和资源管理
六、测试
环境介绍：
基于上一篇Corosync+pacemaker+DRBD+mysql（mariadb）实现高可用（ha）的mysql集群（centos7）博客： http://legehappy.blog.51cto.com/13251607/1976251，突然想到nfs也可以基于这种架构解决单点故障问题，nfs+DRBD+corosync+pacemaker这种架构可实现nfs的多点高可用集群。
系统版本信息：
[root@cml1 ~]# cat/etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root@cml2 ~]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
主机对应关系：
node1：cml1：192.168.5.101
node2：cml2：192.168.5.102
client：cml3：192.168.5.104
配置集群的前提：
（1）时间同步
[root@cml1~]# ntpdate cn.pool.ntp.org
[root@cml2~]# ntpdate cn.pool.ntp.org
（2）主机名互相访问
[root@cml1~]# ssh-keygen
[root@cml1~]# ssh-copy-id cml2
[root@cml1~]# hostname
cml1
[root@cml1~]# cat /etc/hosts
192.168.5.101 cml1 www.cml1.com
192.168.5.102 cml2 www.cml2.com
192.168.5.104 cml3 www.cml3.com
192.168.5.105 cml4 www.cml4.com
（3）是否使用仲裁设备。
Centos7上面不需要使用
二、借助pcs安装与配置corosync和pacemaker（pcs只是一个管理工具）
1、在两节点上执行：
[root@cml1 ~]# yum install -y pacemaker pcspsmisc policycoreutils-python1.
2、两节点上启动pcs并且开机启动：
[root@cml1 ~]# systemctl start pcsd.service[root@cml1 ~]# systemctl enablepcsd.service1.2.
3、两节点上修改用户hacluster的密码（用户已经固定不可以改变）
[root@cml1 ~]# echo redhat | passwd --stdin hacluster1.
4、注册pcs集群主机（默认注册使用用户名hacluster，和密码）：
[root@cml1 corosync]# pcs cluster auth cml25pxl2 ##设置注册那个集群节点cml1: Already authorizedcml2: Already authorized1.2.3.
5、在集群上注册两台集群：
[root@cml1 corosync]# pcs cluster setup--name mycluster cml1 cml2 --force。 ##设置集群1.
6、接下来就在某个节点上已经生成来corosync配置文件：
[root@cml1 corosync]# ls
corosync.conf corosync.conf.example corosync.conf.example.udpu corosync.xml.example uidgid.d
#我们看到生成来corosync.conf配置文件：
7、我们看一下注册进来的文件：
[root@cml1 corosync]# cat corosync.conftotem { version: 2 secauth: off cluster_name: webcluster transport: udpu} nodelist { node { ring0_addr: cml1 nodeid: 1 } node { ring0_addr: cml2 nodeid: 2 }} quorum { provider: corosync_votequorum two_node: 1} logging { to_logfile: yes logfile: /var/log/cluster/corosync.log to_syslog: yes}1.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.26.27.28.29.30.
8、启动集群：
[root@cml1 corosync]# pcs cluster start--allcml1: Starting Cluster...cml2: Starting Cluster...##相当于启动来pacemaker和corosync:[root@cml1 corosync]# ps -ef | grepcorosyncroot 57490 1 1 21:47 ? 00:00:52 corosyncroot 75893 51813 0 23:12 pts/0 00:00:00 grep --color=auto corosync[root@cml1 corosync]# ps -ef | greppacemakerroot 57502 1 0 21:47 ? 00:00:00 /usr/sbin/pacemakerd -fhaclust+ 57503 57502 0 21:47 ? 00:00:03 /usr/libexec/pacemaker/cibroot 57504 57502 0 21:47 ? 00:00:00/usr/libexec/pacemaker/stonithdroot 57505 57502 0 21:47 ? 00:00:01 /usr/libexec/pacemaker/lrmdhaclust+ 57506 57502 0 21:47 ? 00:00:01 /usr/libexec/pacemaker/attrdhaclust+ 57507 57502 0 21:47 ? 00:00:00 /usr/libexec/pacemaker/penginehaclust+ 57508 57502 0 21:47 ? 00:00:01 /usr/libexec/pacemaker/crmdroot 75938 51813 0 23:12 pts/0 00:00:00 grep --color=auto pacemaker1.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.
8、查看集群的状态（显示为no faults就是ok）
[root@cml1 corosync]# corosync-cfgtool -sPrinting ring status.Local node ID 1RING ID 0 id = 192.168.5.101 status = ring 0 active with no faults[root@cml1 corosync]# ssh cml2corosync-cfgtool -sPrinting ring status.Local node ID 2RING ID 0 id = 192.168.5.102 status = ring 0 active with no faults1.2.3.4.5.6.7.8.9.10.11.12.
10、可以查看集群是否有错：
[root@cml1 corosync]# crm_verify -L -V error: unpack_resources: Resource start-up disabled since no STONITH resources have been defined error:unpack_resources: Either configuresome or disable STONITH with the stonith-enabled option error: unpack_resources: NOTE:Clusters with shared data need STONITH to ensure data integrityErrors found during check: config not valid1.2.3.4.5.
##因为我们没有配置STONITH设备，所以我们下面要关闭
11、关闭STONITH设备：
[root@cml1 corosync]# pcs property setstonith-enabled=false[root@cml1 corosync]# crm_verify -L -V[root@cml1 corosync]# pcs property listCluster Properties: cluster-infrastructure: corosync cluster-name: mycluster dc-version: 1.1.16-12.el7_4.2-94ff4df have-watchdog: false stonith-enabled: false1.2.3.4.5.6.7.8.9.三、DRBD安装配置参考之前的博客《DRBD-MYSQL分布式块设备实现高可用》 http://legehappy.blog.51cto.com/13251607/1975804
[root@cml1 drbd.d]# cat nfs.resresource nfs {protocol C;meta-disk internal;device /dev/drbd1;syncer {verify-alg sha1;}net {allow-two-primaries;}on cml1 {disk /dev/sdb1;address 192.168.5.101:7789;}on cml2 {disk /dev/sdb1;address 192.168.5.102:7789;}}1.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.
四、nfs安装与配置：##在node1和node2服务器上配置nfs服务：
[root@cml1 ~]# yum install nfs-utils -y[root@cml1 ~]# systemctl enable nfs-server[root@cml1 ~]# systemctl start nfs-server[root@cml1 ~]# systemctl start rpcbind[root@cml1 ~]# systemctl enable rpcbind1.2.3.4.5.##创建挂载点：
[root@cml1 ~]# cat /etc/exports/nfs_data 192.168.5.0/24(rw,sync)[root@cml1 ~]# mkdir /nfs_data[root@cml2 ~]# cat /etc/exports/nfs_data 192.168.5.0/24(rw,sync)[root@cml2 ~]# mkdir /nfs_data[root@cml1 ~]# systemctl restart nfs-server[root@cml2 ~]# systemctl restart nfs-server##测试查看过载目录：[root@cml3 ~]# showmount -e 192.168.5.101Export list for 192.168.5.101:/nfs_data 192.168.5.0/24[root@cml3 ~]# showmount -e 192.168.5.102Export list for 192.168.5.102:/nfs_data 192.168.5.0/241.2.3.4.5.6.7.8.9.10.11.12.13.14.15.
五、Crmsh安装和资源管理1、安装crmsh：
集群我们可以下载安装crmsh来操作(从github来下载，然后解压直接安装)：只在一个节点安装即可。（但最好选择两节点上安装这样测试时方便点）
[root@cml1 ~]# cd /usr/local/src/You have new mail in /var/spool/mail/root[root@cml1 src]# lsnginx-1.12.0 php-5.5.38.tar.gzcrmsh-2.3.2.tar nginx-1.12.0.tar.gz zabbix-3.2.7.tar.gz[root@cml1 src]# tar -xf crmsh-2.3.2.tar[root@cml1 crmsh-2.3.2]# python setup.pyinstall1.2.3.4.5.6.7.
2、用crmsh来管理：
[root@cml1 ~]# crm help
Help overview for crmsh
Available topics:
Overview Help overview forcrmsh
Topics Available topics
Description Program description
CommandLine Command lineoptions
Introduction Introduction
Interface User interface
Completion Tab completion
Shorthand Shorthand syntax
Features Features
Shadows Shadow CIB usage
Checks Configurationsemantic checks
Templates Configurationtemplates
Testing Resource testing
Security Access ControlLists (ACL)
Resourcesets Syntax: Resourcesets
AttributeListReferences Syntax: Attribute list references
AttributeReferences Syntax: Attribute references
RuleExpressions Syntax: Rule expressions
Lifetime Lifetime parameterformat
Reference Command reference
3、借助crm管理工具配置DRBD+nfs+corosync+pacemaker高可用集群：
##先停掉nfs、drbd服务
[root@cml1 ~]# systemctl stop nfs-server[root@cml1 ~]# systemctl stop drbd[root@cml2 ~]# systemctl stop nfs-server[root@cml2 ~]# systemctl stop drbd [root@cml1 ~]# crmcrm(live)# statusStack: corosyncCurrent DC: cml1 (version1.1.16-12.el7_4.4-94ff4df) - partition with quorumLast updated: Thu Oct 26 08:52:49 2017Last change: Thu Oct 26 08:51:45 2017 byroot via cibadmin on cml1 2 nodes configured0 resources configured Online: [ cml1 cml2 ] No resourcescrm(live)configure# propertystonith-enabled=falsecrm(live)configure# propertyno-quorum-policy=ignorecrm(live)configure# property migration-limit=1 ###表示服务抢占一次不成功就给另一个节点接管crm(live)# configurecrm(live)configure# primitive nfsdrbd ocf:linbit:drbd paramsdrbd_resource=nfs op start timeout=240 op stop timeout=100 op monitorrole=Master interval=20crm(live)configure# ms ms_mysqldrbd nfsdrbd meta master-max=1master-node-max=1 clone-max=2 clone-node-max=1 notify=truecrm(live)configure# verify1.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.
2、添加挂载资源：
crm(live)configure# primitive mystoreocf:heartbeat:Filesystem params device=/dev/drbd1 directory=/nfs_datafstype=ext4 op start timeout=60 op stop timeout=60crm(live)configure# colocationmystore_with_ms_nfsdrbd inf: mystore ms_mysqldrbd:Mastercrm(live)configure# orderms_mysqld_befor_mystore Mandatory: ms_mysqldrbd mystorecrm(live)configure# verifycrm(live)configure# commitcrm(live)configure# cdcrm(live)# statusStack: corosyncCurrent DC: cml1 (version1.1.16-12.el7_4.4-94ff4df) - partition with quorumLast updated: Thu Oct 26 21:08:41 2017Last change: Thu Oct 26 21:08:38 2017 byroot via cibadmin on cml1 2 nodes configured3 resources configured Online: [ cml1 cml2 ] Full list of resources: Master/Slave Set: ms_nfsdrbd [nfsdrbd] Masters: [ cml2 ] Slaves: [ cml1 ] mystore (ocf::heartbeat:Filesystem): Started cml2[root@cml2 ~]# df -THFilesystem Type Size Used Avail Use% Mounted on/dev/mapper/centos-root xfs 19G 6.7G 13G 36% /devtmpfs devtmpfs 501M 0 501M 0% /devtmpfs tmpfs 512M 278M 234M 55% /dev/shmtmpfs tmpfs 512M 27M 486M 6% /runtmpfs tmpfs 512M 0 512M 0% /sys/fs/cgroup/dev/sda1 xfs 521M 161M 361M 31% /boottmpfs tmpfs 103M 0 103M 0% /run/user/0/dev/drbd1 ext4 11G 69M 9.9G 1% /nfs_data1.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.26.27.28.29.30.31.32.33.
3、添加nfs_server
crm(live)configure# primitive nfs_serversystemd:nfs-server op start timeout=100 interval=0 op stop timeout=100interval=0crm(live)configure# verifycrm(live)configure# colocation nfs_server_with_mystore inf:nfs_server mystorecrm(live)configure# order mystore_befor_nfsMandatory: mystore nfs_servercrm(live)configure# shownode 1: cml1 \ attributesstandby=offnode 2: cml2 \ attributesstandby=offprimitive mystore Filesystem \ paramsdevice="/dev/drbd1" directory="/nfs_data" fstype=ext4 \ opstart timeout=60 interval=0 \ opstop timeout=60 interval=0primitive nfs_server systemd:nfs-server \ opstart timeout=100 interval=0 \ opstop timeout=100 interval=0primitive nfsdrbd ocf:linbit:drbd \ paramsdrbd_resource=nfs \ opstart timeout=240 interval=0 \ opstop timeout=100 interval=0 \ opmonitor role=Master interval=20 timeout=30 \ opmonitor role=Slave interval=30 timeout=30ms ms_nfsdrbd nfsdrbd \ metamaster-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=trueorder ms_nfsdrbd_befor_mystore Mandatory: ms_nfsdrbdmystoreorder mystore_befor_nfs Mandatory: mystorenfs_servercolocation mystore_with_ms_nfsdrbd inf:mystore ms_nfsdrbd:Mastercolocation nfs_server_with_mystore inf:nfs_server mystoreproperty cib-bootstrap-options: \ have-watchdog=false\ dc-version=1.1.16-12.el7_4.4-94ff4df\ cluster-infrastructure=corosync\ cluster-name=webcluster\ stonith-enabled=false\ no-quorum-policy=ignore\ migration-limit=11.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.26.27.28.29.30.31.32.33.34.35.36.
4、添加虚拟vip：
crm(live)configure# primitive vipocf:heartbeat:IPaddr params ip=192.168.5.200 op monitor interval=20 timeout=20on-fail=restartcrm(live)configure# verifycrm(live)configure# colocation vip_with_nfsinf: vip nfs_servercrm(live)configure# verifycrm(live)configure# shownode 1: cml1 \ attributesstandby=offnode 2: cml2 \ attributesstandby=offprimitive mystore Filesystem \ paramsdevice="/dev/drbd1" directory="/nfs_data" fstype=ext4 \ opstart timeout=60 interval=0 \ opstop timeout=60 interval=0primitive nfs_server systemd:nfs-server \ opstart timeout=100 interval=0 \ opstop timeout=100 interval=0primitive nfsdrbd ocf:linbit:drbd \ paramsdrbd_resource=nfs \ opstart timeout=240 interval=0 \ opstop timeout=100 interval=0 \ opmonitor role=Master interval=20 timeout=30 \ opmonitor role=Slave interval=30 timeout=30primitive vip IPaddr \ paramsip=192.168.5.200 \ opmonitor interval=20 timeout=20 on-fail=restartms ms_nfsdrbd nfsdrbd \ metamaster-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=trueorder ms_nfsdrbd_befor_mystore Mandatory:ms_nfsdrbd mystoreorder mystore_befor_nfs Mandatory: mystorenfs_servercolocation mystore_with_ms_nfsdrbd inf:mystore ms_nfsdrbd:Mastercolocation nfs_server_with_mystore inf:nfs_server mystorecolocation vip_with_nfs inf: vip nfs_serverproperty cib-bootstrap-options: \ have-watchdog=false\ dc-version=1.1.16-12.el7_4.4-94ff4df\ cluster-infrastructure=corosync\ cluster-name=webcluster\ stonith-enabled=false\ no-quorum-policy=ignore\ migration-limit=1crm(live)configure# commit1.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.20.21.22.23.24.25.26.27.28.29.30.31.32.33.34.35.36.37.38.39.40.41.
5、查看节点状态：
crm(live)# statusStack: corosyncCurrent DC: cml1 (version1.1.16-12.el7_4.4-94ff4df) - partition with quorumLast updated: Thu Oct 26 21:14:37 2017Last change: Thu Oct 26 21:14:22 2017 byroot via cibadmin on cml1 2 nodes configured5 resources configured Online: [ cml1 cml2 ] Full list of resources: Master/Slave Set: ms_nfsdrbd [nfsdrbd] Masters: [ cml2 ] Slaves: [ cml1 ] mystore (ocf::heartbeat:Filesystem): Started cml2 nfs_server (systemd:nfs-server): Started cml2 vip (ocf::heartbeat:IPaddr): Started cml21.2.3.4.5.6.7.8.9.10.11.12.13.14.15.16.17.18.19.
六、测试：[root@cml2 ~]# df -TH
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/centos-root xfs 19G 6.7G 13G 36% /
devtmpfs devtmpfs 501M 0 501M 0% /dev
tmpfs tmpfs 512M 278M 234M 55% /dev/shm
tmpfs tmpfs 512M 27M 486M 6% /run
tmpfs tmpfs 512M 0 512M 0% /sys/fs/cgroup
/dev/sda1 xfs 521M 161M 361M 31% /boot
tmpfs tmpfs 103M 0 103M 0% /run/user/0
/dev/drbd1 ext4 11G 69M 9.9G 1% /nfs_data
[root@cml2 ~]# ip addr
2: ens34:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen1000
link/ether 00:0c:29:5a:c5:ee brd ff:ff:ff:ff:ff:ff
inet 192.168.5.102/24 brd 192.168.5.255 scope global ens34
valid_lft forever preferred_lft forever
inet 192.168.5.200/24brd 192.168.5.255 scope global secondary ens34
valid_lft forever preferred_lft forever
###vip已经在cml2主机上了
[root@cml3 ~]# showmount -e 192.168.5.200
Export list for 192.168.5.200:
/nfs_data 192.168.5.0/24
[root@cml3 ~]# showmount -e 192.168.5.200
Export list for 192.168.5.200:
/nfs_data 192.168.5.0/24
[root@cml3 ~]# mkdir /nfs
[root@cml3 ~]# mount -t nfs192.168.5.200:/nfs_data/ /nfs
[root@cml3 ~]# df -TH
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/centos-root xfs 19G 6.6G 13G 35% /
devtmpfs devtmpfs 503M 0 503M 0% /dev
tmpfs tmpfs 513M 0 513M 0% /dev/shm
tmpfs tmpfs 513M 14M 500M 3% /run
tmpfs tmpfs 513M 0 513M 0% /sys/fs/cgroup
/dev/sda1 xfs 521M 131M 391M 25% /boot
tmpfs tmpfs 103M 0 103M 0% /run/user/0
192.168.5.200:/nfs_datanfs4 11G 69M 9.9G 1% /nfs
###挂载后也是11G证明是drbd挂载到nfs_data目录在挂载过来的空间-----------------------------------©著作权归作者所有：来自51CTO博客作者legehappy的原创作品，如需转载，请注明出处，否则将追究法律责任nfs+DRBD+corosync+pacemaker 实现高可用（ha）的nfs集群https://blog.51cto.com/legehappy/1976565
