基于 heartbeat v2 crm 实现HA高可用性的 LAMP+wordpress
环境配置：
OS version: CentOS release 6.7 (Final) Soft version: heartbeat-2.1.4-11.el5.x86_64.rpm heartbeat-devel-2.1.4-11.el5.x86_64.rpm heartbeat-pils-2.1.4-9.el5.x86_64.rpm heartbeat-stonith-2.1.4-11.el5.x86_64.rpm libltdl3-1.5.26-alt11.x86_64.rpm 高可用节点: node 1 192.168.2.216/24 node1.test.com 高可用节点: node 2 192.168.2.217/24 node2.test.com VIP: node 3 192.168.2.218/24
关闭iptables,selinux
关闭iptables、selinux,例如：[root@php-server www]service iptables stop [root@php-server www]chkconfig iptables off [root@php-server www]getenforce 0
安装并同步各个节点的时间,例如
[root@lvs-dr ~]# yum install -y ntpdate.x86_64 [root@lvs-dr ~]# ntpdate cn.ntp.org.cn
一、修改主机名
[root@node1 ~]# more /etc/hosts hosts hosts.allow hosts.deny [root@node1 ~]# more /etc/hosts 127.0.0.1 localhost node1 node1.test.com localhost4 localhost4.localdomain4 ::1 localhost node1 node1.test.com localhost6 localhost6.localdomain6 [root@node1 ~]# more /etc/sysconfig/network NETWORKING=yes HOSTNAME=node1
二、安装apahce
[root@node1 ~]# yum install -y httpd [root@node1 ~]# vim /var/www/html/index.html [root@node1 ~]# service httpd restart Stopping httpd: [ OK ] Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using node1.test.com for ServerName [ OK ] [root@node1 ~]# curl -I http://192.168.2.216 HTTP/1.1 200 OK Date: Sat, 13 Aug 2016 16:32:38 GMT Server: Apache/2.2.15 (CentOS) Last-Modified: Sat, 13 Aug 2016 16:30:22 GMT ETag: "260270-f-539f683fbfea1" Accept-Ranges: bytes Content-Length: 15 Connection: close Content-Type: text/html; charset=UTF-8
三、SSH免密钥登录
[root@node1 ~]# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Created directory '/root/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: f0:33:40:f4:f1:83:3d:7d:1e:5b:67:a2:a8:08:44:b3 root@node1.test.com The key's randomart image is: +--[ RSA 2048]----+ | o.o . | | . + . = . | | E o o = . + +| | . + + + *.| | . S . . o | | . . + | | . . | | | | | +-----------------+ [root@node1 ~]# ssh-copy-id 192.168.2.217 The authenticity of host '192.168.2.217 (192.168.2.217)' can't be established. RSA key fingerprint is bc:71:16:97:d8:ce:a0:3e:8b:4a:b9:da:ca:bd:dd:07. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.2.217' (RSA) to the list of known hosts. root@192.168.2.217's password: Now try logging into the machine, with "ssh '192.168.2.217'", and check in: .ssh/authorized_keys to make sure we haven't added extra keys that you weren't expecting. [root@node1 ~]# ssh root@192.168.2.217 Last login: Sun Aug 14 00:24:50 2016 from 192.168.2.106 [root@node2 ~]#
四、安装heartbeat
[root@node1 ~]# wget http://mirrors.aliyun.com/epel/6/x86_64/epel-release-6-8.noarch.rpm [root@node1 ~]# rpm -ivh epel-release-6-8.noarch.rpm warning: epel-release-6-8.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY Preparing... ########################################### [100%] 1:epel-release ########################################### [100%] #直接安装rpm heartbeat会报依赖错误，这里我们先解决依赖关系 [root@node1 ~]#yum -y install perl-TimeDate net-snmp-libs libnet PyXML #再安装就可以了 [root@node1 heartbeat]# rpm -ivh *.rpm --nodeps --force warning: heartbeat-2.1.4-11.el5.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 217521f6: NOKEY warning: heartbeat-pils-2.1.4-9.el5.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 192a7d7d: NOKEY Preparing... ########################################### [100%] 1:heartbeat-pils ########################################### [ 25%] 2:heartbeat-stonith ########################################### [ 50%] 3:heartbeat warning: /etc/ha.d/shellfuncs created as /etc/ha.d/shellfuncs.rpmnew ########################################### [ 75%] 4:heartbeat-devel ########################################### [100%] [root@node1 heartbeat]# ll /etc/ha.d/ total 24 -rwxr-xr-x 1 root root 745 Mar 21 2010 harc drwxr-xr-x 2 root root 4096 Aug 14 17:14 rc.d -rw-r--r-- 1 root root 692 Mar 21 2010 README.config drwxr-xr-x 2 root root 4096 Aug 14 17:14 resource.d -rw-r--r-- 1 root root 7864 Mar 21 2010 shellfuncs [root@node1 ha.d]# cp -r /usr/share/doc/heartbeat-2.1.4/ /etc/ha.d/ [root@node1 ha.d]# ll total 28 -rwxr-xr-x 1 root root 745 Mar 21 2010 harc drwxr-xr-x 2 root root 4096 Aug 14 04:24 heartbeat-2.1.4 drwxr-xr-x 2 root root 4096 Aug 14 04:20 rc.d -rw-r--r-- 1 root root 692 Mar 21 2010 README.config drwxr-xr-x 2 root root 4096 Aug 14 04:20 resource.d -rw-r--r-- 1 root root 7864 Mar 21 2010 shellfuncs [root@node1 ha.d]# cp ./heartbeat-2.1.4/{ha.cf,authkeys,haresources} ./ [root@node1 ha.d]# chmod 600 authkeys /etc/ha.d目录下： #ha.cf :主配置文件，定义各节点上的heartbeat HA集群的基本属性； #authkeys :集群内节点间彼此传递消息时使用加密算法及密钥； #haresources :为heartbeat v1的提供资源管理器配置接口: v1版本专用的配置接口 [root@node1 ha.d]# ll total 52 -rw------- 1 root root 666 Aug 14 04:27 authkeys -rw-r--r-- 1 root root 10770 Aug 14 04:29 ha.cf -rwxr-xr-x 1 root root 745 Mar 21 2010 harc -rw-r--r-- 1 root root 5970 Aug 14 04:54 haresources drwxr-xr-x 2 root root 4096 Aug 14 04:24 heartbeat-2.1.4 drwxr-xr-x 2 root root 4096 Aug 14 22:39 rc.d -rw-r--r-- 1 root root 692 Mar 21 2010 README.config drwxr-xr-x 2 root root 4096 Aug 14 22:39 resource.d -rw-r--r-- 1 root root 7864 Mar 21 2010 shellfuncs
这里要注意的是：如何在网络中我们期望的节点集群成为我们所需要的节点，在集群中信息不能随便传递，而心跳节点是基于组播地址传递的，如果别人也装了heartbeat也连接到这个组播地址上来，这都不安全，基于这种情况，我们各节点这间信息传递是需要认证的，这种认证基于HMAC
配置heartbeat，它的配置文件在/etc/ha.d/的目录下，但是安装完程序之后这个目录下没有这个配置文件，只有/usr/share/doc/heartbeat-2.1.4/目录下有ha.cf的主配置文件样本，复制到/etc下修改配置文件即可使用；还有一个authkeys的认证文件，这个文件就是我们各节点认证时所保存的认证密码和认证机制，所以这个文件的权限至关重要，必须是600，否则启动不了服务；第三个haresources，定义资源时需要资源管理器来读取这个文件，所以这个也得有；
[root@node1 ha.d]# vim authkeys auth 2 2 sha1 be45617e827e7a7d
[root@node1 ha.d]# vim ha.cf debugfile /var/log/ha-debug keepalive 2 deadtime 30 warntime 10 udpport 694 mcast eth0 225.10.0.1 694 1 0 auto_failback on node node1.test.com node node2.test.com crm on ping 192.168.2.217 compression bz2 compression_threshold 2 #logfile /var/log/ha-log #日志文件，正常日志信息记录到哪去的 #keepalive 2 #每隔2s发送一次心跳信息的，单位是秒，毫秒用ms #deadtime 30 #隔多长时间探测到对方不在线就kill掉的时间间隔 #warntime 10 #警告时间 #udpport 694 #mcast eth0 225.10.0.1 694 1 0 #定义组播地址 #auto_failback on #开启故障转回功能 #node node1.test.com #定义两个节点 #node node2.test.com #crm on #启用crm功能 #ping 192.168.2.218 #ping节点 #compression bz2 #压缩格式 #compression_threshold 2 #表示小于2K时不压缩传输
[root@node1 ha.d]# scp -p ha.cf haresources authkeys node2:/etc/ha.d/ The authenticity of host 'node2 (192.168.2.217)' can't be established. RSA key fingerprint is bc:71:16:97:d8:ce:a0:3e:8b:4a:b9:da:ca:bd:dd:07. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'node2' (RSA) to the list of known hosts. ha.cf 100% 11KB 10.5KB/s 00:00 haresources 100% 5970 5.8KB/s 00:00 authkeys 100% 666 0.7KB/s 00:00
在资源管理器的配置文件中定义；/etc/ha.d/haresources，在/etc/ha.d/resource.d下有各种资源类型，当在资源配置文件中定义时就会调用这里的资源类型来运行相应的程序；
[root@node1 ha.d]# node1.test.com 192.168.2.218/24/eth0/192.168.255.255 httpd [root@node1 ha.d]# service heartbeat start [root@node1 ha.d]# service heartbeat start logd is already running Starting High-Availability services: Done.
碰到一些包依赖的错误，都是小问题就不一一举例了，缺什么补什么，还有就是刚启动服务，机器就重启，懵逼了。。。报错信息如下：
[root@node1 ~]# Message from syslogd@node1 at Aug 15 01:32:04 ... heartbeat: [1439]: EMERG: Rebooting system. Reason: /usr/lib64/heartbeat/cibatu Connection closed by foreign host.
网上有说改hearbeat文件的用户和组，没用~~，后注释ha.cf的#crm on 恢复。
[root@node1 ha.d]# service heartbeat restart Stopping High-Availability services: Done. Waiting to allow resource takeover to complete: Done. Starting High-Availability services: 2016/08/14_18:08:55 INFO: Resource is stopped Done.
检测运行状态,所有资源在node1上 ，node2 上只有heartbeat服务
[root@node1 ha.d]# ifconfig eth0 Link encap:Ethernet HWaddr 00:0C:29:46:B1:B7 inet addr:192.168.2.216 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe46:b1b7/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2522 errors:0 dropped:0 overruns:0 frame:0 TX packets:4289 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:320135 (312.6 KiB) TX bytes:2902096 (2.7 MiB) eth0:0 Link encap:Ethernet HWaddr 00:0C:29:46:B1:B7 inet addr:192.168.2.218 Bcast:10.10.10.254 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) [root@node1 ha.d]# ps -ef |grep httpd root 2635 1 0 18:10 ? 00:00:00 /usr/sbin/httpd apache 2640 2635 0 18:10 ? 00:00:00 /usr/sbin/httpd apache 2641 2635 0 18:10 ? 00:00:00 /usr/sbin/httpd apache 2642 2635 0 18:10 ? 00:00:00 /usr/sbin/httpd apache 2643 2635 0 18:10 ? 00:00:00 /usr/sbin/httpd apache 2644 2635 0 18:10 ? 00:00:00 /usr/sbin/httpd apache 2645 2635 0 18:10 ? 00:00:00 /usr/sbin/httpd apache 2646 2635 0 18:10 ? 00:00:00 /usr/sbin/httpd apache 2647 2635 0 18:10 ? 00:00:00 /usr/sbin/httpd root 2900 1592 0 18:12 pts/0 00:00:00 grep httpd [root@node1 ha.d]# pa aux |grep heartbeat -bash: pa: command not found [root@node1 ha.d]# ps aux |grep heartbeat root 2338 0.0 1.4 69608 14364 ? SLs 18:10 0:00 heartbeat: master control process nobody 2341 0.0 0.7 62876 7632 ? SL 18:10 0:00 heartbeat: FIFO reader nobody 2342 0.0 0.7 63004 7760 ? SL 18:10 0:00 heartbeat: write: mcast eth0 nobody 2343 0.0 0.7 63004 7760 ? SL 18:10 0:00 heartbeat: read: mcast eth0 nobody 2344 0.0 0.7 63004 7760 ? SL 18:10 0:00 heartbeat: write: ping 192.168.2.217 nobody 2345 0.0 0.7 63004 7760 ? SL 18:10 0:00 heartbeat: read: ping 192.168.2.217 root 2904 0.0 0.0 103312 876 pts/0 S+ 18:13 0:00 grep heartbeat [root@node2 ha.d]# ifconfig eth0 Link encap:Ethernet HWaddr 00:0C:29:2D:5B:EC inet addr:192.168.2.217 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe2d:5bec/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:11766 errors:0 dropped:0 overruns:0 frame:0 TX packets:5494 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3603815 (3.4 MiB) TX bytes:837732 (818.0 KiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:355 errors:0 dropped:0 overruns:0 frame:0 TX packets:355 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:49398 (48.2 KiB) TX bytes:49398 (48.2 KiB) [root@node2 ha.d]# netstat -nlptu|grep 80 tcp 0 0 :::80 :::* LISTEN 3198/httpd [root@node2 ha.d]# ps -ef |grep httpd root 3198 1 0 18:11 ? 00:00:00 /usr/sbin/httpd apache 3200 3198 0 18:11 ? 00:00:00 /usr/sbin/httpd apache 3201 3198 0 18:11 ? 00:00:00 /usr/sbin/httpd apache 3202 3198 0 18:11 ? 00:00:00 /usr/sbin/httpd apache 3203 3198 0 18:11 ? 00:00:00 /usr/sbin/httpd apache 3204 3198 0 18:11 ? 00:00:00 /usr/sbin/httpd apache 3205 3198 0 18:11 ? 00:00:00 /usr/sbin/httpd apache 3206 3198 0 18:11 ? 00:00:00 /usr/sbin/httpd apache 3207 3198 0 18:11 ? 00:00:00 /usr/sbin/httpd root 3217 1811 0 18:14 pts/0 00:00:00 grep httpd [root@node2 ha.d]#
测试网页：
停止node1节点上的heartbeat后
[root@node2 ha.d]# ifconfig eth0 Link encap:Ethernet HWaddr 00:0C:29:2D:5B:EC inet addr:192.168.2.217 Bcast:192.168.2.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe2d:5bec/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:12689 errors:0 dropped:0 overruns:0 frame:0 TX packets:6338 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3701251 (3.5 MiB) TX bytes:980188 (957.2 KiB) eth0:0 Link encap:Ethernet HWaddr 00:0C:29:2D:5B:EC inet addr:192.168.2.218 Bcast:192.168.2.254 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:413 errors:0 dropped:0 overruns:0 frame:0 TX packets:413 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:57576 (56.2 KiB) TX bytes:57576 (56.2 KiB) [root@node2 ha.d]# ps -ef |grep httpd root 4452 1 0 18:21 ? 00:00:00 /usr/sbin/httpd apache 4454 4452 0 18:21 ? 00:00:00 /usr/sbin/httpd apache 4455 4452 0 18:21 ? 00:00:00 /usr/sbin/httpd apache 4456 4452 0 18:21 ? 00:00:00 /usr/sbin/httpd apache 4457 4452 0 18:21 ? 00:00:00 /usr/sbin/httpd apache 4458 4452 0 18:21 ? 00:00:00 /usr/sbin/httpd apache 4459 4452 0 18:21 ? 00:00:00 /usr/sbin/httpd apache 4460 4452 0 18:21 ? 00:00:00 /usr/sbin/httpd apache 4461 4452 0 18:21 ? 00:00:00 /usr/sbin/httpd root 4468 1811 0 18:21 pts/0 00:00:00 grep httpd [root@node2 ha.d]# ps -ef |grep heartbeat root 4022 1 0 18:20 ? 00:00:00 heartbeat: master control process nobody 4024 4022 0 18:20 ? 00:00:00 heartbeat: FIFO reader nobody 4025 4022 0 18:20 ? 00:00:00 heartbeat: write: mcast eth0 nobody 4026 4022 0 18:20 ? 00:00:00 heartbeat: read: mcast eth0 nobody 4027 4022 0 18:20 ? 00:00:00 heartbeat: write: ping 192.168.2.216 nobody 4028 4022 0 18:20 ? 00:00:00 heartbeat: read: ping 192.168.2.216 root 4470 1811 0 18:21 pts/0 00:00:00 grep heartbeat
切换到node2上面了：
当node1节点停止后，node2的节点就会顶上去，成为主节点，提供高可用服务，httpd内容现实node1.test.com和node2.test.com只是为了测试，加以区别。
附上node1、node2 ha-debug
[root@node1 ha.d]# tail -f /var/log/ha-debug heartbeat[2338]: 2016/08/14_18:16:23 info: killing HBWRITE process 2342 with signal 15 heartbeat[2338]: 2016/08/14_18:16:23 info: killing HBREAD process 2343 with signal 15 heartbeat[2338]: 2016/08/14_18:16:23 info: killing HBWRITE process 2344 with signal 15 heartbeat[2338]: 2016/08/14_18:16:23 info: killing HBREAD process 2345 with signal 15 heartbeat[2338]: 2016/08/14_18:16:23 info: Core process 2342 exited. 5 remaining heartbeat[2338]: 2016/08/14_18:16:23 info: Core process 2341 exited. 4 remaining heartbeat[2338]: 2016/08/14_18:16:23 info: Core process 2345 exited. 3 remaining heartbeat[2338]: 2016/08/14_18:16:23 info: Core process 2344 exited. 2 remaining heartbeat[2338]: 2016/08/14_18:16:23 info: Core process 2343 exited. 1 remaining heartbeat[2338]: 2016/08/14_18:16:23 info: node1.test.com Heartbeat shutdown complete. [root@node2 ha.d]# tail -f /var/log/ha-debug IPaddr[4294]:2016/08/14_18:21:13 INFO: Success INFO: Success ResourceManager[4196]:2016/08/14_18:21:13 debug: /etc/ha.d/resource.d/IPaddr 192.168.2.218/24/eth0/192.168.2.254 start done. RC=0 ResourceManager[4196]:2016/08/14_18:21:13 info: Running /etc/init.d/httpd start ResourceManager[4196]:2016/08/14_18:21:13 debug: Starting /etc/init.d/httpd start Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using node2.test.com for ServerName [ OK ] ResourceManager[4196]:2016/08/14_18:21:13 debug: /etc/init.d/httpd start done. RC=0 heartbeat[4022]: 2016/08/14_18:21:23 info: Local Resource acquisition completed. (none) heartbeat[4022]: 2016/08/14_18:21:23 info: local resource transition completed.
安装 heartbeat CRM LAMP WordPress
[root@node1 ha.d]# yum install mysql-server php php-mysql -y [root@node1 ha.d]# vim /var/www/html/info.php [root@node1 ha.d]# service httpd restart Stopping httpd: [ OK ] Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using node1.test.com for ServerName [ OK ]
打卡node1节点httpd，已经配置好php了
测试mysql
[root@node1 ha.d]# mysql Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.1.73 Source distribution Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | test | +--------------------+ 3 rows in set (0.00 sec) mysql>
nfs服务器上：
[root@php-server ~]# mkdir /data/mysql
node1上:
[root@node1 ha.d]# more /etc/my.cnf [mysqld] datadir=/data/ mount -t nfs 192.168.2.216:/data/mysql /data /usr/bin/mysql_install_db --user=mysql --datadir=/data/ socket=/var/lib/mysql/mysql.sock user=mysql # Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0 [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid [root@node1 ha.d]# service mysqld start Starting mysqld: [ OK ] [root@node1 ha.d]#mysql mysql> CREATE DATABASE `wordpress` CHARACTER SET utf8 COLLATE utf8_general_ci; Query OK, 1 row affected (0.00 sec) mysql> GRANT ALL PRIVILEGES ON wordpress.* TO 'wordpress'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | test | | wordpress | +--------------------+ 4 rows in set (0.00 sec)
安装 heartbeat-gui
[root@node1 ~]# yum -y install pygtk2-libglade xorg-x11-xauth [root@node1 ~]# rpm -ivh heartbeat-gui-2.1.4-12.el6.x86_64.rpm [root@node1 ~]# echo "password" | passwd --stdin hacluster
