keepalived的工作原理解析以及安装使用

一、keepalived

keepalived是集群管理中保证集群高可用的一个服务软件，其功能类似于heartbeat，用来防止单点故障。

keepalived官网http://www.keepalived.org

二、keepalived工作原理

keepalived软件主要是通过VRRP（Virtual Router RedundancyProtocol虚拟路由器冗余协议）实现高可用功能的。

虚拟路由冗余协议，可以认为是实现路由器高可用的协议，即将N台提供相同功能的路由器组成一个路由器组，这个组里面有一个master和多个backup，master上面有一个对外提供服务的vip（该路由器所在局域网内其他机器的默认路由为该vip），master会发组播，当backup收不到vrrp包时就认为master宕掉了，这时就需要根据VRRP的优先级来选举一个backup当master。这样的话就可以保证路由器的高可用了。

keepalived主要有三个模块，分别是core、check和vrrp。core模块为keepalived的核心，负责主进程的启动、维护以及全局配置文件的加载和解析。check负责健康检查，包括常见的各种检查方式。vrrp模块是来实现VRRP协议的。

三、keepalived实现nginx服务高可用

1、实验环境

 

  

IP规划 ：

keepalived1—192.168.137.121

keepalived2—192.168.137.122

VIP—192.168.137.100

高可用主机上安装keepalived作为HA，再安装nginx作为web代理服务器 ，后端tomcat（实验环境下偷个懒，就不配nginx反向代理到tomcat了，直接nginx配一样的页面）

2、安装

2.1、nginx安装

为了实验方便，采用yum安装方式

[root@keepalived1 ~]#rpm -ivh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
[root@keepalived1 ~]#yum install nginx -y
[root@keepalived1 ~]#nginx    //启动

 

验证：

这里修改两个web内容不一样，是为了区分我们的流量访问的哪台keepalived。生产环境中主机提供的内容必须一致，需要nginx代理到相同的后端服务器tomcat或者服务器挂载共享磁盘

[root@keepalived1 ~]# echo web1 > /usr/share/nginx/html/index.html
[root@keepalived2 ~]# echo web2 > /usr/share/nginx/html/index.html

 

 

 

 现在web1和web2都正常工作

 

2.2、keepalived安装

官网下载：https://www.keepalived.org/software/keepalived-1.4.5.tar.gz

方法一：yum安装

[root@keepalived1 ~]#yum install keepalived -y

/etc/keepalived
/etc/keepalived/keepalived.conf     #keepalived服务主配置文件
/etc/rc.d/init.d/keepalived         #服务启动脚本
/etc/sysconfig/keepalived
/usr/bin/genhash
/usr/libexec/keepalived
/usr/sbin/keepalived

方法二：编译安装

yum安装编译所需依赖
yum install -y gcc glibc openssl openssl-devel libnl libnl-devel libnfnetlink-devel
[root@keepalived1 tools]$ tar -zxvf keepalived-1.4.5.tar.gz
[root@keepalived1 tools]$cd keepalived-1.4.5/

编译
[root@keepalived1 keepalived-1.4.5]$ ./configure --prefix=/usr/local/keepalived
[root@keepalived1 keepalived-1.4.5]$ make && make install

安装完成，复制配置文件模板到/etc/keepalived
mkdir /etc/keepalived
cp /tools/keepalived-1.4.5/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
cp /tools/keepalived-1.4.5/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

复制服务启动脚本：
cp /tools/keepalived-1.4.5/keepalived/etc/init.d/keepalived /etc/init.d/
chmod +x /etc/init.d/keepalived

centos7的话还需要改/lib/systemd/system/keepalived.service

将里面的：

EnvironmentFile=-/usr/local/keepalived/etc/sysconfig/keepalived
ExecStart=/usr/local/keepalived/sbin/keepalived $KEEPALIVED_OPTIONS

修改成：

EnvironmentFile=/etc/sysconfig/keepalived
ExecStart=/sbin/keepalived $KEEPALIVED_OPTIONS

然后重新加载service

systemctl daemon-reload

创建命令软连接：
ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin/keepalived

常用的选项
keepalived -D -f /etc/keepalived/keepalived.conf
-D   将日志输出到message日志,默认日志也在message
-f    是指定配置文件

3、改变keepalived服务的日志路径：

修改/etc/sysconfig/keepalived

把KEEPALIVED_OPTIONS="-D" 修改为：KEEPALIVED_OPTIONS="-D -d -S 0"

[root@keepalived2 ~]# vim /etc/sysconfig/keepalived 
# Options for keepalived. See `keepalived --help' output and keepalived(8) and
# keepalived.conf(5) man pages for a list of all options. Here are the most
# common ones :
#
# --vrrp               -P    Only run with VRRP subsystem.
# --check              -C    Only run with Health-checker subsystem.
# --dont-release-vrrp  -V    Dont remove VRRP VIPs & VROUTEs on daemon stop.
# --dont-release-ipvs  -I    Dont remove IPVS topology on daemon stop.
# --dump-conf          -d    Dump the configuration data.
# --log-detail         -D    Detailed log messages.
# --log-facility       -S    0-7 Set local syslog facility (default=LOG_DAEMON)
#

KEEPALIVED_OPTIONS="-D -d -S 0"                      //-S 是syslog的facility,0表示放在local 0

 

在/etc/rsyslog.conf 末尾添加

[root@keepalived2 ~]# vim /etc/rsyslog.conf
local0.*                                                /var/log/keepalived.log

 重启syslog

[root@keepalived2 log]# service rsyslog restart 

 

重启keepalived后就可以看到日志在/var/log/keepalived.log下了。

注意：

centos7还需修改/lib/systemd/system/keepalived.service 文件：

因为centos 7使用systemctl，通过systemctl调用service，所以需要修改/lib/systemd/system/keepalived.service文件。

将里面的：

EnvironmentFile=-/usr/local/keepalived/etc/sysconfig/keepalived
ExecStart=/usr/local/keepalived/sbin/keepalived $KEEPALIVED_OPTIONS

修改成：

EnvironmentFile=/etc/sysconfig/keepalived
ExecStart=/sbin/keepalived $KEEPALIVED_OPTIONS

 然后重新加载service

systemctl daemon-reload

配置完成，查看日志

[root@keepalived2 log]# systemctl restart rsyslog
[root@keepalived2 log]# systemctl restart keepalived
[root@keepalived2 log]# tail -f /var/log/keepalived.log
Feb 23 11:40:30 keepalived2 Keepalived_vrrp[66778]: Sending gratuitous ARP on eno16777736 for 192.168.137.100
Feb 23 11:40:30 keepalived2 Keepalived_vrrp[66778]: Sending gratuitous ARP on eno16777736 for 192.168.137.100
Feb 23 11:40:30 keepalived2 Keepalived_vrrp[66778]: Sending gratuitous ARP on eno16777736 for 192.168.137.100
Feb 23 11:40:30 keepalived2 Keepalived_vrrp[66778]: Sending gratuitous ARP on eno16777736 for 192.168.137.100
Feb 23 11:40:35 keepalived2 Keepalived_vrrp[66778]: Sending gratuitous ARP on eno16777736 for 192.168.137.100
Feb 23 11:40:35 keepalived2 Keepalived_vrrp[66778]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eno16777736 for 192.168.137.100
Feb 23 11:40:35 keepalived2 Keepalived_vrrp[66778]: Sending gratuitous ARP on eno16777736 for 192.168.137.100
Feb 23 11:40:35 keepalived2 Keepalived_vrrp[66778]: Sending gratuitous ARP on eno16777736 for 192.168.137.100
Feb 23 11:40:35 keepalived2 Keepalived_vrrp[66778]: Sending gratuitous ARP on eno16777736 for 192.168.137.100
Feb 23 11:40:35 keepalived2 Keepalived_vrrp[66778]: Sending gratuitous ARP on eno16777736 for 192.168.137.100

4、配置文件含义

默认配置文件中各配置的含义

! Configuration File for keepalived

global_defs {                    //全局配置
   notification_email {        //定义报警邮件地址
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc    //定义发送邮件的地址
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL         //定义路由标识信息，相同局域网唯一
   vrrp_skip_check_adv_addr        //检查vrrp报文中的所有地址比较耗时，设置此标志的意思是如果接收的到报文和上一个报文来至同一个路由器，则不执行检查。默认是跳过检查
   vrrp_strict      // #严格遵守vrrp协议，下面这些功能将会禁止：1.   0 VIP   2. unicast(单播) peers    3. vrrp 版本2的ipv6功能
   vrrp_garp_interval 0  //小数类型，单位秒，在一个网卡上每组gratuitous arp消息之间的延迟时间，默认为0，一个发送的消息=n组 arp报文
   vrrp_gna_interval 0     //小数类型，单位秒， 在一个网卡上每组na消息之间的延迟时间，默认为0
}

vrrp_instance VI_1 {      //定义实例
    state MASTER            //初始状态，MASTER|BACKUP ,一旦有其他机器加入，将会举行选举，具有最高优先级的机器将会成为MASTER,所以这个条目的并不重要
    interface eth0            //指定该实例用户vrrp的网卡，用于发送vrrp
    virtual_router_id 51    //指定VRRP实例ID，范围是0-255.同一个组要一致
    priority 100               // 指定优先级，优先级高的将成为MASTER。
    advert_int 1              // 指定发送VRRP通告的间隔。单位是秒。
    authentication {        //指定认证方式。PASS简单密码认证(推荐),AH:IPSEC认证(不推荐)。密码" 最多8位
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {    //设备之间使用的虚拟ip地址
        192.168.200.16
        192.168.200.17
        192.168.200.18
    }    

5、使用keepalived实现nginx高可用

配置高可用

1、修改配置文件

keepalived1

! Configuration File for keepalived
 
global_defs {
   router_id nginx1
}
 
vrrp_script chk_nginx {
   #检查nginx的脚本,需要我们自己定义，下面讲到
   script "/etc/keepalived/nginx_check.sh"            
   #检查时间间隔,这个时间不要超过脚本的执行时间，否则会报“Track script chk_nginx is being timed out, expect idle - skipping run”
   interval 2                                         
   #脚本执行失败则优先级减20
   weight -20
   #表示两次失败才算失败
   fall 2　　　　　　　　　　　　　　　　　　　　　　　　 
}
#  weight： 
#1. 如果脚本执行成功(退出状态码为0)，weight大于0，则priority增加。 
#2. 如果脚本执行失败(退出状态码为非0)，weight小于0，则priority减少。 
#3. 其他情况下，priority不变。
 
vrrp_instance VI_1 {
    state MASTER
    interface eno16777736
    virtual_router_id 11
    mcast_src_ip 192.168.137.121
    priority 100
    #设置为不抢占。默认是抢占的，当高优先级的机器恢复后，会抢占低优先级的机器成为MASTER，而不抢占，则允许低优先级的机器继续成为MASTER，即使高优先级的机器已经上线。如果要使用这个功能，则初始化状态必须为BACKUP。
   # nopreempt               
    #指定发送VRRP通告的间隔。单位是秒。
    advert_int 1            
    authentication {
        auth_type PASS
        auth_pass 1111
    }
#对应上面的检查脚本，使之生效
    track_script {
       chk_nginx
    }
#vip的地址
    virtual_ipaddress {
        192.168.137.100
    }
}

keepalived2

! Configuration File for keepalived
 
global_defs {
   router_id nginx2
}
 
vrrp_script chk_nginx {
   script "/etc/keepalived/nginx_check.sh"            
   interval 2 
}
 
vrrp_instance VI_1 {
    #修改初始状态为备机
    state BACKUP
    interface eno16777736
    virtual_router_id 11
    mcast_src_ip 192.168.137.122
    #修改优先级为小于正常状态下master的优先级，大于降低了权重之后的优先级
    priority 90
   # nopreempt               
    advert_int 1            
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    track_script {
       chk_nginx
    }
    virtual_ipaddress {
        192.168.137.100
    }
}

 

2、两台主机都设置nginx检查脚本

说明：

keepalived主备切换方式：①根据vrrp的优先级，优先级高的为主，优先级低的为备     ②vrrp探测主节点的keepalived挂掉时备节点主动升级为master角色

两种方式的检查脚本不同

①根据vrrp的优先级，定义脚本检查nginx状态，如果状态异常则放回脚本执行失败 返回码为1。这个时候服务优先级根据配置调整

vim /etc/keepalived/nginx_check.sh

#!/bin/sh
A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ]
then
  /usr/sbin/nginx
  sleep 1
  A2=`ps -C nginx --no-header |wc -l`
  if [ $A2 -eq 0 ]
  then
    exit 1
  fi
fi

②vrrp探测主节点的keepalived挂掉时备节点主动升级为master角色，脚本中当检查到nginx状态异常后将执行杀死keepalived服务

#!/bin/sh
A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ]
then
  /usr/sbin/nginx
  sleep 1
  A2=`ps -C nginx --no-header |wc -l`
  if [ $A2 -eq 0 ]
  then
    systemctl stop keepalived
  fi
fi

 授权可执行权限 chmod +x /etc/keepalived/nginx_check.sh

6、防止脑裂

1）关闭SELinux

setenforce 0    #设置为宽容模式

但这样只在本次生效，重启服务器后将失效。如果要永久关闭，还需要修改配置文件：

sed -i 's/=enforcing/=disabled/g' /etc/sysconfig/selinux

 

2）防火墙放通

centos 防火墙有两种管理方式firewall， iptables两者不能同时开启

防火墙开启的情况下，我们需要加入一条配置：

iptables

编辑vim /etc/sysconfig/iptables

-A INPUT -p vrrp -j ACCEPT

注意：

添加规则一定不要在

-A INPUT -j REJECT --reject-with icmp-host-prohibited

之后，一定要加在其前面。

配置完之后reload

service iptables reload

Firewalld防火墙配置

centos7 默认防火墙firewall

开启vrrp 协议

主备都运行下面的命令

firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0  --protocol vrrp -j ACCEPT

firewall-cmd --reload

7、验证高可用

首先验证vip在主节点生效

验证master上的nginx关闭，master自动执行检查脚本并启动nginx

 

验证故障切换，通过修改配置文件模拟nginx挂了起不来

脚本①脚本执行返回错误，执行优先级-20，VIP转移到从节点

脚本②检查脚本将停掉keepalived，vip转移到从节点

  

访问vip，由nginx2提供服务

 验证keepalived主从切换成功。

采用的检查脚本是当nginx状态异常后停主节点启用备节点