高可用keepalived基础

官方文档：

https://keepalived.org/doc/
http://keepalived.org/documentation.html

1）用户空间核心组件：
vrrp stack：VIP消息通告
checkers：监测real server
system call：实现 vrrp 协议状态转换时调用脚本的功能
SMTP：邮件组件
IPVS wrapper：生成IPVS规则
Netlink Reflector：网络接口
WatchDog：监控进程

2）控制组件：提供keepalived.conf 的解析器，完成Keepalived配置

3）IO复用器：针对网络目的而优化的自己的线程抽象

4）内存管理组件：为某些通用的内存管理功能（例如分配，重新分配，发布等）提供访问权限

各节点时间必须同步：ntp, chrony

关闭防火墙及SELinux

各节点之间可通过主机名互相通信：非必须

建议使用/etc/hosts文件实现：非必须

各节点之间的root用户可以基于密钥认证的ssh服务完成互相通信：非必须

Keepalived 相关文件

软件包名：keepalived
主程序文件：/usr/sbin/keepalived
主配置文件：/etc/keepalived/keepalived.conf
配置文件示例：/usr/share/doc/keepalived/
Unit File：/lib/systemd/system/keepalived.service
Unit File的环境配置文件：
/etc/sysconfig/keepalived CentOS
/etc/default/keepalived Ubuntu

注意：CentOS 7 上有 bug，可能有下面情况出现
systemctl restart keepalived #新配置可能无法生效
systemctl stop keepalived;systemctl start keepalived #无法停止进程，需要 kill停止

Keepalived 包安装

#CentOS
[root@centos ~]# yum -y install keepalived
#ubuntu
[root@ubuntu1804 ~]# apt -y install keepalived

编译安装

[root@centos7 ~]# yum -y install gcc curl openssl-devel libnl3-devel net-snmp-devel

[root@centos7 ~]# wget https://keepalived.org/software/keepalived-2.0.20.tar.gz

[root@centos7 ~]# tar xvf keepalived-2.0.20.tar.gz -C /usr/local/src
[root@centos7 ~]# cd /usr/local/src/keepalived-2.0.20/

#选项--disable-fwmark 可用于禁用iptables规则,可访止VIP无法访问,无此选项默认会启用ipatbles
规则
[root@centos7 keepalived-2.0.20]#./configure --prefix=/usr/local/keepalived #--disable-fwmark

[root@centos7 keepalived-2.0.20]# make && make install
[root@centos7 keepalived-2.0.20]# cd
[root@centos7 ~]# /usr/local/keepalived/sbin/keepalived -v

#默认会自动生成unit文件
[root@centos7 ~]# cat /usr/lib/systemd/system/keepalived.service
[Unit]
Description=LVS and VRRP High Availability Monitor
After=network-online.target syslog.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/run/keepalived.pid
KillMode=process
EnvironmentFile=-/usr/local/keepalived/etc/sysconfig/keepalived
ExecStart=/usr/local/keepalived/sbin/keepalived $KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
[Install]

WantedBy=multi-user.target

#默认无法启动
[root@centos7 ~]# systemctl start keepalived.service

#查看日志，可以看到是因为缺少配置文件导致无法启动
[root@centos7 ~]# journalctl -xe

#创建配置文件
[root@centos7 ~]# mkdir /etc/keepalived
[root@centos7 ~]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf
/etc/keepalived

#再次启动成功
[root@centos7 ~]# systemctl enable --now keepalived.service

#默认生成iptables规则,无法访问VIP,编译时可以加--disable-fwmark禁用生成iptables规则
[root@centos7 ~]# iptables -vnL

[root@centos7 ~]# vim /etc/keepalived/keepalived.conf
#注释下面一行
#vrrp_strict

#重启动不生效，有bug
[root@centos7 ~]# systemctl restart keepalived.service

#无法关闭进程
[root@centos7 ~]# systemctl stop keepalived.service
[root@centos7 ~]# ps aux | grep keepalived

#强制杀进程
[root@centos7 ~]# killall keepalived
[root@centos7 ~]# systemctl start keepalived.service

KeepAlived 配置说明

配置文件

/etc/keepalived/keepalived.conf

配置文件组成

GLOBAL CONFIGURATION
Global definitions：定义邮件配置，route_id，vrrp配置，多播地址等
VRRP CONFIGURATION
VRRP instance(s)：定义每个vrrp虚拟路由器
LVS CONFIGURATION
Virtual server group(s)
Virtual server(s)：LVS集群的VS和RS

帮助

man keepalived.conf

全局配置

#/etc/keepalived/keepalived.conf
global_defs {
notification_email {
root@localhost #keepalived 发生故障切换时邮件发送的目标邮箱，可以按行区分写
多个
root@xx.com
xxx@qq.com
}
notification_email_from keepalived@localhost #发邮件的地址
smtp_server 127.0.0.1 #邮件服务器地址
smtp_connect_timeout 30 #邮件服务器连接timeout
router_id ka1.example.com #每个keepalived主机唯一标识，建议使用当前主机名，但多节点重名
不影响
vrrp_skip_check_adv_addr #对所有通告报文都检查，会比较消耗性能，启用此配置后，如果收到的
通告报文和上一个报文是同一个路由器，则跳过检查，默认值为全检查
vrrp_strict #严格遵守VRRP协议,启用此项后以下状况将无法启动服务:1.无VIP地址 2.配置了单播邻
居 3.在VRRP版本2中有IPv6地址，开启动此项并且没有配置vrrp_iptables时会自动开启iptables防火
墙规则，默认导致VIP无法访问,建议不加此项配置
vrrp_garp_interval 0 #gratuitous ARP messages 报文发送延迟，0表示不延迟
vrrp_gna_interval 0 #unsolicited NA messages （不请自来）消息发送延迟
vrrp_mcast_group4 224.0.0.18 #指定组播IP地址范围：224.0.0.0到239.255.255.255,默认
值：224.0.0.18
vrrp_iptables #此项和vrrp_strict同时开启时，则不会添加防火墙规则,如果无配置
vrrp_strict项,则无需启用此项配置
}

配置虚拟路由器

vrrp_instance <STRING> { #<String>为vrrp的实例名,一般为业务名称
配置参数
......
}
#配置参数：
state MASTER|BACKUP#当前节点在此虚拟路由器上的初始状态，状态为MASTER或者BACKUP
interface IFACE_NAME #绑定为当前虚拟路由器使用的物理接口，如：eth0,bond0,br0,可以和VIP不在一个网卡
virtual_router_id VRID #每个虚拟路由器惟一标识，范围：0-255，每个虚拟路由器此值必须唯一，否则服务无法启动，同属一个虚拟路由器的多个keepalived节点必须相同,务必要确认在同一网络中此值必须唯一
priority 100 #当前物理节点在此虚拟路由器的优先级，范围：1-254，值越大优先级越高,每个keepalived主机节点此值不同
advert_int 1 #vrrp通告的时间间隔，默认1s
authentication { #认证机制
auth_type AH|PASS #AH为IPSEC认证(不推荐),PASS为简单密码(建议使用)
auth_pass <PASSWORD> #预共享密钥，仅前8位有效，同一个虚拟路由器的多个keepalived节点必
须一样
}
virtual_ipaddress { #虚拟IP,生产环境可能指定上百个IP地址
<IPADDR>/<MASK> brd <IPADDR> dev <STRING> scope <SCOPE> label <LABEL>
192.168.200.100 #指定VIP，不指定网卡，默认为eth0,注意：不指定/prefix,默认为/32
192.168.200.101/24 dev eth1 #指定VIP的网卡，建议和interface指令指定的岗卡不在一个网卡
192.168.200.102/24 dev eth2 label eth2:1 #指定VIP的网卡label
}
track_interface { #配置监控网络接口，一旦出现故障，则转为FAULT状态实现地址转移
eth0
eth1
…
}

启用keepalived日志功能
范例：

[root@ka1 ~]# vim /etc/sysconfig/keepalived
KEEPALIVED_OPTIONS="-D -S 6"

[root@ka1 ~]# vim /etc/rsyslog.conf
local6.* /var/log/keepalived.log

[root@ka1 ~]# systemctl restart keepalived.service rsyslog.service
[root@ka1 ~]# tail -f /var/log/keepalived.log

实现独立子配置文件

当生产环境复杂时， /etc/keepalived/keepalived.conf 文件中内容过多，不易管理，可以将不同集群的配置，比如：不同集群的VIP配置放在独立的子配置文件中利用include指令可以实现包含子配置文件

格式:

include /path/file

范例:

[root@ka1-centos8 ~]# mkdir /etc/keepalived/conf.d/
[root@ka1-centos8 ~]# vim /etc/keepalived/keepalived.conf
global_defs {
    notification_email {
        xxxx@qq.com
    }
    notification_email_from xxxx@qq.com
    smtp_server 127.0.0.1
    smtp_connect_timeout 30
    router_id ka1.longxuan.vip
    vrrp_skip_check_adv_addr
    vrrp_garp_interval 0
    vrrp_gna_interval 0
}
include /etc/keepalived/conf.d/*.conf

#将VRRP相关配置放在子配置文件中
[root@ka1-centos8 ~]# vim /etc/keepalived/conf.d/cluster1.conf

实现master/slave的 Keepalived 单主架构

MASTER配置

[root@ka1-centos8 ~]# vim /etc/keepalived/keepalived.conf
global_defs {
    notification_email {
        root@localhost #keepalived 发生故障切换时邮件发送的对象，可以按行区分写多个
    }
    notification_email_from keepalived@localhost
    smtp_server 127.0.0.1
    smtp_connect_timeout 30
    router_id ka1.example.com
    vrrp_skip_check_adv_addr #所有报文都检查比较消耗性能，此配置为如果收到的报文和上一个报文是同一个路由器则跳过检查报文中的源地址
    #vrrp_strict #严格遵守VRRP协议,禁止状况:1.无VIP地址,2.配置了单播邻居,3.在VRRP版本2中有IPv6地址
    vrrp_garp_interval 0 #ARP报文发送延迟
    vrrp_gna_interval 0 #消息发送延迟
    vrrp_mcast_group4 224.0.0.18 #默认组播IP地址，可指定组播范围：224.0.0.0到239.255.255.255
}
vrrp_instance VI_1 {
    state MASTER #在另一个结点上为BACKUP
    interface eth0
    virtual_router_id 66 #每个虚拟路由器必须唯一，同属一个虚拟路由器的多个keepalived节点必须相同
    priority 100 #在另一个结点上为80
    advert_int 1
    authentication {
        auth_type PASS #预共享密钥认证，同一个虚拟路由器的keepalived节点必须一样
        auth_pass 123456
    }
    virtual_ipaddress {
        172.31.0.188 dev eth0 label eth0:0
    }
}

BACKUP配置

#配置文件和master基本一致，只需修改三行
[root@ka2-centos8 ~]# vim /etc/keepalived/keepalived.conf
global_defs {
    notification_email {
        root@localhost
    }
    notification_email_from keepalived@localhost
    smtp_server 127.0.0.1
    smtp_connect_timeout 30
    router_id ka2.example.com #修改此行
    vrrp_skip_check_adv_addr
    #vrrp_strict
    vrrp_garp_interval 0
    vrrp_gna_interval 0
    vrrp_mcast_group4 224.0.0.18
}
vrrp_instance VI_1 {
    state BACKUP #修改此行
    interface eth0
    virtual_router_id 66
    priority 80 #修改此行
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        172.31.0.188 dev eth0 label eth0:0
    }
}

抓包观察

tcpdump -i eth0 -nn host 224.0.0.18

抢占模式和非抢占模式

非抢占模式 nopreempt
默认为抢占模式preempt，即当高优先级的主机恢复在线后，会抢占低先级的主机的master角色，造成
网络抖动，建议设置为非抢占模式 nopreempt ，即高优先级主机恢复后，并不会抢占低优先级主机的
master角色
非抢点模块下,如果原主机down机, VIP迁移至的新主机, 后续也发生down时,仍会将VIP迁移回原主机

注意：要关闭 VIP抢占，必须将各 keepalived 服务器state配置为BACKUP

#ha1主机配置
vrrp_instance VI_1 {
    state BACKUP #都为BACKUP
    interface eth0
    virtual_router_id 66
    priority 100 #优先级高
    advert_int 1
    nopreempt #添加此行，都为nopreempt
  
#ha2主机配置
vrrp_instance VI_1 {
    state BACKUP #都为BACKUP
    interface eth0
    virtual_router_id 66
    priority 80 #优先级低
    advert_int 1
    nopreempt #添加此行，都为nopreempt

抢占延迟模式 preempt_delay

抢占延迟模式，即优先级高的主机恢复后，不会立即抢回VIP，而是延迟一段时间（默认300s）再抢回VIP

preempt_delay # #指定抢占延迟时间为#s，默认延迟300s

注意：需要各keepalived服务器state为BACKUP,并且不要启用 vrrp_strict

#ha1主机配置
vrrp_instance VI_1 {
    state BACKUP #都为BACKUP
    interface eth0
    virtual_router_id 66
    priority 100 #优先级高
    advert_int 1
    preempt_delay 10 #抢占延迟模式，默认延迟300s

#ha2主机配置
vrrp_instance VI_1 {
   state BACKUP #都为BACKUP
   interface eth0
   virtual_router_id 66
   priority 80 #优先级低
   advert_int 1

VIP单播配置

默认keepalived主机之间利用多播相互通告消息，会造成网络拥塞，可以替换成单播，减少网络流量

注意：启用 vrrp_strict 时，不能启用单播

#在所有节点vrrp_instance语句块中设置对方主机的IP，建议设置为专用于对应心跳线网络的地址，而非使
用业务网络
unicast_src_ip <IPADDR> #指定发送单播的源IP
unicast_peer {
<IPADDR> #指定接收单播的对方目标主机IP
......
}
#启用 vrrp_strict时，不能启用单播,否则服务无法启动,并在messages文件中记录下面信息

范例：

#master 主机配置
[root@ka1-centos8 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    notification_email {
        acassen@firewall.loc
        failover@firewall.loc
        sysadmin@firewall.loc
    }
    notification_email_from Alexandre.Cassen@firewall.loc
    smtp_server 192.168.200.1
    smtp_connect_timeout 30
    router_id ka1.magedu.org
    vrrp_skip_check_adv_addr
    #vrrp_strict
    vrrp_garp_interval 0
    vrrp_gna_interval 0
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 66
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        172.31.0.188/24 dev eth0 label eth0:1
   }
    unicast_src_ip 172.31.0.8 #本机IP
    unicast_peer{
        172.31.0.18 #指向对方主机IP
        #172.31.0.28 #如果有多个keepalived,再加其它节点的IP
    }
}

#slave 主机配置
[root@ka2-centos8 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    notification_email {
        acassen@firewall.loc
        failover@firewall.loc
        sysadmin@firewall.loc
    }
  notification_email_from Alexandre.Cassen@firewall.loc
  smtp_server 192.168.200.1
  smtp_connect_timeout 30
  router_id ka2.magedu.org
  vrrp_skip_check_adv_addr
  #vrrp_strict
  vrrp_garp_interval 0
  vrrp_gna_interval 0
}
vrrp_instance VI_1 {
    state SLAVE
    interface eth0
    virtual_router_id 66
    priority 80
    advert_int 1
    authentication {
       auth_type PASS
       auth_pass 123456
  }
  virtual_ipaddress {
       172.31.0.188/24 dev eth0 label eth0:1
  }
  unicast_src_ip 172.31.0.18 #本机IP
  unicast_peer {
      172.31.0.8 #指向对方主机IP
  }
}

抓包

root@centos ~]# tcpdump -i eth0 -nn src host 172.31.0.8 and dst host 172.31.0.18

Keepalived 通知脚本配置

当keepalived的状态变化时，可以自动触发脚本的执行，比如：发邮件通知用户
默认以用户keepalived_script身份执行脚本，如果此用户不存在，以root执行脚本
可以用下面指令指定脚本执行用户的身份

global_defs {
    ......
    script_user <USER>
    ......
}

通知脚本类型

当前节点成为主节点时触发的脚本

notify_master <STRING>|<QUOTED-STRING>

当前节点转为备节点时触发的脚本

notify_backup <STRING>|<QUOTED-STRING>

当前节点转为“失败”状态时触发的脚本

notify_fault <STRING>|<QUOTED-STRING>

通用格式的通知触发机制，一个脚本可完成以上三种状态的转换时的通知

notify <STRING>|<QUOTED-STRING>

当停止VRRP时触发的脚本

notify_stop <STRING>|<QUOTED-STRING>

脚本的调用方法

在 vrrp_instance VI_1 语句块的末尾加下面行

notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"

创建通知脚本

[root@ka1-centos8 ~]#cat /etc/keepalived/notify.sh
#!/bin/bash
#
contact='root@qq.com'
notify() {
    mailsubject="$(hostname) to be $1, vip floating"
    mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
    echo "$mailbody" | mail -s "$mailsubject" $contact
}
case $1 in
master)
    notify master
    ;;
backup)
    notify backup
    ;;
fault)
    notify fault
    ;;
*)
    echo "Usage: $(basename $0) {master|backup|fault}"
    exit 1
    ;;
esac

邮件配置

# QQ 邮箱
[root@centos8 ~]# vim /etc/mail.rc
#在最后面添加下面行
set from=36258964@qq.com
set smtp=smtp.qq.com
set smtp-auth-user=36258964@qq.com
set smtp-auth-password=dfadfadgg3ee
set smtp-auth=login
set ssl-verify=ignore

范例：163 邮箱配置

[root@centos8 ~]# vim /etc/mail.rc
set from=xxx@163.com #之前设置好的邮箱地址
set smtp=smtp.163.com #邮件服务器
set smtp-auth-user=xxx@163.com #之前设置好的邮箱地址
set smtp-auth-password=yyy #授权码
set smtp-auth=login #默认login即可

范例：发送测试邮件

[root@centos8 ~]# yum -y install mailx
[root@centos8 ~]# echo "Test Mail"| mail -s Warning root@qq.com

实战案例：实现 Keepalived 状态切换的通知脚本

#在所有 keepalived节点配置如下
[root@ka1-centos8 ~]# cat /etc/keepalived/notify.sh
#!/bin/bash
#
contact='root@qq.com'
notify() {
    mailsubject="$(hostname) to be $1, vip floating"
    mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
    echo "$mailbody" | mail -s "$mailsubject" $contact
}
case $1 in
master)
    notify master
    ;;
backup)
    notify backup
    ;;
fault)
    notify fault
    ;;
*)
    echo "Usage: $(basename $0) {master|backup|fault}"
    exit 1
    ;;
esac

[root@ka1-centos8 ~]# chmod a+x /etc/keepalived/notify.sh
[root@ka1-centos8 ~]# vim /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
    ......
    virtual_ipaddress {
        172.31.0.188 dev eth0 label eth0:1
    }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"
}

#模拟master故障
[root@ka1-centos8 ~]#killall keepalived

# 查看邮箱收到邮件