keepalived+haproxy实现高可用
- 实验环境:
- 2台centos 6.5作为keepalived+haproxy的高可用，3台centos6.5配置httpd作为后端server,haproxy的轮询采用rr调度算法。vip:192.168.8.199
- 在ha1,ha2上安装keepalived+haproxy
- 在rs1,rs2,rs3上安装httpd，并更改默认首页。
- 验证httpd是否正常，在ha1上运行：
- haproxy的配置：
- keepalived配置
- 结合状态检测，当haproxy出现故障时候自动切换。
- 利用vrrp_script来监控业务状态，进行权重减值。既然要监控haproxy的状态,我们可以直接调用haproxy的状态页面来进行确认。其调用方法为:不能定义在frontend字段里面。可以定义在其他字段
- 直接利用脚本，间隔运行监控haproxy的状态，当haproxy状态异常的时候，直接停用keepalived，其状态恢复后，启用keepalived，这样也可以实现业务层面的异常后切换。并且这个还需要识别是人工停用，还是手动停用的keepalived，人工由于配置等原因停用的keepalived则，不处理。保持其状态。
ha1:eth1:192.168.8.41,keepalived+haproxy
ha3:eth1:192.168.8.43,keepalived+haproxy
rs1:192.168.8.21.httpd
rs2:192.168.8.22.httpd
rs3:192.168.8.23.httpd
yum -y install keepalived haproxy
yum -y install httpd
关闭rs1,rs2,rs3的防火墙，或者开发80端口，这里做实验方便一些，直接关闭防火墙。
service iptables stop
更改各自的首页，以便于识别是由哪个 rs提供服务的。
rs1: echo "rs1" > /var/www/html/index.html
rs2:echo "rs2" > /var/www/html/index.html
rs3:echo "rs3" > /var/www/html/index.html
启动httpd服务，在rs1,rs2,rs3上:service httpd start
curl http://192.168.8.21 看是否获取到rs1的内容，同样
curl http://192.168.8.22
curl http://192.168.8.23
上面的配置，保证了后端服务器提供服务的能力，这里我们要配置haproxy作为http反向代理服务器的配置。
在ha1上
| cat /etc/haproxy/haproxy.cfg 你会看到默认配置文件。 首先更改日志到本地日志系统里面去。由于centos 6.5使用的是rsyslog，所以vi /etc/sysconfig/rsyslog 更改为： SYSLOGD_OPTIONS="-c 5 -r" 然后重启rsyslog [root@ha3 ~]# service rsyslog restart 关闭系统日志记录器： [确定] 启动系统日志记录器：-r option only supported in compatibility modes 0 to 2 - ignored 提示错误，将5改成2，使用兼容模式。 然后在/etc/rsyslog.conf里面添加： local2.* /var/log/haproxy.log进去，并且要启 # Provides UDP syslog reception $ModLoad imudp $UDPServerRun 514这两个参数。 用重启rsyslog ：service rsyslog restart .更改后配置文件如下： [root@ha1 ~]# cat /etc/haproxy/haproxy.cfg global chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon log 127.0.0.1 local2 # turn on stats unix socket stats socket /var/lib/haproxy/stats
#--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000
#--------------------------------------------------------------------- # main frontend which proxys to the backends #--------------------------------------------------------------------- frontend http mode http bind *:80 option httpclose option logasap option dontlognull capture request header Host len 20 capture request header Referer len 60 default_backend servers backend servers balance roundrobin server websrv1 192.168.8.21:80 check maxconn 2000 server websrv2 192.168.8.22:80 check maxconn 2000 server websrv3 192.168.8.23:80 check maxconn 2000 [root@ha1 ~]#service haproxy start 启动haproxy，则用curl 127.0.0.1 多用几次则可以发现在调度。 | [root@ha1 ~]# cat /etc/haproxy/haproxy.cfg #--------------------------------------------------------------------- # Example configuration for a possible web application. See the # full configuration options online. # # http://haproxy.1wt.eu/download/1.4/doc/configuration.txt # #---------------------------------------------------------------------
#--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global # to have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2
chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon
# turn on stats unix socket stats socket /var/lib/haproxy/stats
#--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000
#--------------------------------------------------------------------- # main frontend which proxys to the backends #--------------------------------------------------------------------- frontend main *:5000 acl url_static path_beg -i /static /images /javascript /stylesheets acl url_static path_end -i .jpg .gif .png .css .js
use_backend static if url_static default_backend app
#--------------------------------------------------------------------- # static backend for serving up images, stylesheets and such #--------------------------------------------------------------------- backend static balance roundrobin server static 127.0.0.1:4331 check
#--------------------------------------------------------------------- # round robin balancing between the various backends #--------------------------------------------------------------------- backend app balance roundrobin server app1 127.0.0.1:5001 check server app2 127.0.0.1:5002 check server app3 127.0.0.1:5003 check server app4 127.0.0.1:5004 check
[root@ha1 ~]# |
在ha3上同样配置如上内容。
可以使用scp命令直接把配置文件传送过去
ha1:scp /etc/haprxoy/haproxy.conf 192.168.8.43:/etc/haproxy
然后手动修改ha3上面的rsyslog的相关配置。
手动使用curl检测是否配置OK。。
通过上面的配置，两个haproxy已经配置OK，后端rs也准备就绪。把两个haproxy配置成高可用。
编辑ha1的/etc/keepalived/keepalived.conf
由于我们这里只有一个实例，并且没有使用lvs的功能，故vrrp_group和vitrual_server都不需要定义。
配置文件：
| ha1: | ha3: |
| [root@ha1 ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived
global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL }
vrrp_instance keepalived { state MASTER interface eth1 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.8.199/24 } }
[root@ha1 ~]# | [root@ha3 ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived
global_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL }
vrrp_instance keepalived { state BACKUP interface eth1 virtual_router_id 51 priority 90 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.8.199/24 } }
[root@ha3 ~]# |
结合日志和curl的工具，确定最先开始由ha1提供服务，将ha1,keepalived停掉后切换到ha3上面了。证明高可用成功。
要实现haproxy出现故障时自动切换，两种方案：一种利用vrrp_script 来监控业务状态。进行权重减值。另外一种，直接利用脚本监控haproxy的状态，OK后就开启keepalived，失败后就停用keepalived 。
| backend private_monitoring stats enable stats uri /admin?stats stats refresh 5s | 说明： stats enable Enable statistics reporting with default settings May be used in sections : defaults | frontend | listen | backend yes | no | yes | yes |
开启haproxy的监控页面。将上面参数添加到backend里面去。然后重启haproxy，利用curl 127.0.0.1:80/admin?stats就可以查看是否正常开启监控页面。这里监控页面已经打开，那么编写vrrp_scrpit来监控业务状态。
curl -m 5 -I localhost:80/admin?stats -o /dev/null -s 用来检测是否正常，正常情况下#?的值为0，异常为>0
在 keepalived的global_def之后紧跟着定义vrrp_script 脚本
| vrrp_script check_haproxy { script "/usr/bin/curl -m 5 -I localhost:80/admin?stats -o /dev/null -s" interval 6 weight -20 } | interval 6 代表，代表6秒运行一次。 weight -20 表示当script返回值为非0时，权重减去对应的值。 如果weight为正值，表示script返回值=0时，权值加值。 |
| 上面的优先级不会不停的降低。 |
|
在keepalived vrrp_instance keepalived 里面添加如下内容：
track_script
{
check_haproxy
}
即可。这样当haproxy挂掉后，会自动切换到backup上面去。当master的haproxy挂掉后，自动恢复到master。
| #!/bin/bash touchfile=/tmp/keepalived_check_status.txt haproxy_http_stats=`curl -I -m5 -o /dev/numm localhost:80/admin?stats -s -w %{http_stats] ` if [ "$haproxy_http_stats != 200 ] ; then ps -A | grep haproxy | grep -v grep | grep keepalived> /dev/null && service keepalived stop && touch $touchfile else ps -A | grep haproxy | grep -v grep | grep keepalived> /dev/null || [ -f $touchfile ] && service keepalived start && rm -r $touchfile fi 然后放到定时任务里面 ，每分钟执行一次即可。
|
|
