Keepalived+Tengine高可用负载均衡部署实践

Keepalived+Tengine高可用负载均衡部署实践

介绍

概述
Tengine是由淘宝网发起的Web服务器项目。它在Nginx的基础上，针对大访问量网站的需求，添加了很多高级功能和特性。Tengine的性能和稳定性已经在大型的网站如淘宝、天猫商城等得到了很好的检验。它的最终目标是打造一个高效、稳定、安全、易用的Web平台。它有很多良好的特性：支持百万级高并发，动态模块加载（DSO），强大的负载均衡能力，会话保持模块，主动健康检查，根据服务器状态自动上线下线，以及动态解析upstream中出现的域名等一系列强大的功能；

Keepalived是一个免费开源的，用C编写的类似于layer3, 4 & 7交换机制软件，具备我们平时说的第3层、第4层和第7层交换机的功能。主要提供loadbalancing（负载均衡）和 high-availability（高可用）功能，负载均衡实现需要依赖Linux的虚拟服务内核模块（ipvs），而高可用是通过VRRP协议实现多台机器之间的故障转移服务。

所以Tengine很适合用来做七层的负载均衡，而用Keepalived来解决单点故障实现高可用；阿里云的七层负载便是用Keepalived+Tengine实现的，这两个会是很好的组合；Keepalived有主备、主主模式；这里使用主备模式：即一台出现故障，VIP实现漂移到另外一台自动接管服务。

1：环境

CentOS7.9      1C1G       10.0.0.10       Keepalived+Tengine-master
CentOS7.9      1C1G       10.0.0.11       Keepalived+Tengine-backup
CentOS7.9      1C1G       10.0.0.12       nginx_server_1
CentOS7.9      1C1G       10.0.0.13       nginx_server_2

2：配置

1：配置主机名
hostnamectl set-hostname keepalived-tengine-master
hostnamectl set-hostname keepalived-tengine-backup
hostnamectl set-hostname nginx_server_1
hostnamectl set-hostname nginx_server_2

2：时间同步
yum install -y chrony
systemctl enable chronyd --now
chronyc sources -v

3：安装所需软件
keepalived-tengine-master
keepalived-tengine-backup
#两台主机执行

yum install -y keepalived
systemctl start keepalived.service && systemctl enable keepalived
# keepalived-tengine-master 服务器配置
! Configuration File for keepalived

global_defs {
   router_id master                #运行Keepalived服务器的标识,自定义；
}

vrrp_script chk_nginx {                 #定义一个外部脚本
    script "/etc/keepalived/chk_nginx.sh"    #脚本的路径
    interval 1
    weight 2
}

vrrp_instance VI_1 {           #实例名称为VI_1,相同实例的备节点名字要和这个相同
    state MASTER               #状态为MASTER,备节点状态需要为BACKUP
    interface eth0             #通信接口为ens33 此参数备节点设置和主节点相同
    virtual_router_id 51      #虚拟路由ID;一组集群ID号必须一样
    priority 150                  #权重,BACKUP不能高于MASTER
    advert_int 1                  #检测时间间隔
    authentication {
        auth_type PASS         #认证类型
        auth_pass 1111         #认证密码,同一集群密码要一样
    }
    virtual_ipaddress {
        10.0.0.100
    }                                 #配置的虚拟ip，掩码24，并绑定网卡ens33接口，别名为ens33:2 
                                    #此参数备节点设置和主节点设置相同
}
track_script {                  #定义状态跟踪，名称为vrrp_script中定义的
        chk_nginx
}
# keepalived-tengine-backup服务器配置
! Configuration File for keepalived

global_defs {
   router_id buckup                #运行Keepalived服务器的标识,自定义；
}

vrrp_script chk_nginx {                 #定义一个外部脚本
    script "/etc/keepalived/chk_nginx.sh"    #脚本的路径
    interval 1
    weight 2
}

vrrp_instance VI_1 {           #实例名称为VI_1,相同实例的备节点名字要和这个相同
    state BACKUP               #状态为MASTER,备节点状态需要为BACKUP
    interface eth0             #通信接口为ens33 此参数备节点设置和主节点相同
    virtual_router_id 51      #虚拟路由ID;一组集群ID号必须一样
    priority 100                  #权重,BACKUP不能高于MASTER
    advert_int 1                  #检测时间间隔
    authentication {
        auth_type PASS         #认证类型
        auth_pass 1111         #认证密码,同一集群密码要一样
    }
    virtual_ipaddress {
        10.0.0.100
    }
                                    #此参数备节点设置和主节点设置相同
}
track_script {                  #定义状态跟踪，名称为vrrp_script中定义的
        chk_nginx
}

在主服务器上除了可以让keepalived挂掉后自动漂移外，还可以自定义一个检测nginx服务是否存活的脚本，若是nginx无法访问那么自动关闭Keepalived让其自动漂移到备用服务器，在主配置文件的脚本如下：

vim /etc/keepalived/chk_nginx.sh 
#!/bin/bash
NGINX_ACTIVE=$(ss -lnt | grep "80" | wc -l)
if [ $NGINX_ACTIVE -eq 0 ];then
        systemctl stop keepalived
    else
        exit 0
fi
eof

添加执行权限：chmod +x /etc/keepalived/chk_nginx.sh

编译安装
yum install -y gcc gcc-c++ autoconf automake pcre pcre-devel openssl openssl-devel
wget http://tengine.taobao.org/download/tengine-2.3.3.tar.gz
tar xf tengine-2.3.3.tar.gz && cd tengine-2.3.3
./configure --with-http_sub_module --with-http_stub_status_module --with-http_gzip_static_module
make && make install

	关于动态模块：
	如果你想要编译官方模块为动态模块，你需要在configure的时候加上类似这样的指令(–with-http_xxx_module),./configure –help可以看到更多的细节.
	如果只想要安装官方模块为动态模块(不安装Nginx)，那么就只需要configure之后，执行 make dso_install命令.
	动态加载模块的个数限制为128个.
	如果已经加载的动态模块有修改，那么必须重起Tengine才会生效.
	只支持HTTP模块
	Tengine默认将安装在/usr/local/nginx目录，nginx -m 可以查看模块列表。
	
3：配置Tengine，实现对后端web集群的负载
编辑  vim /usr/local/nginx/conf/nginx.conf配置文件插入以下参数
http{

	upstream master {
        # simple round-robin
        server 10.0.0.12:80;
        server 10.0.0.13:80;
    }      

    server {
        listen 80;
        server_name localhost;
        location / {
            proxy_pass http://master;
            proxy_set_header Host  $host;  #匹配请求头对应后其端服务器
        }
        location /status {     #状态监控
                stub_status;
        }
    }
}

4：启动并检测是否成功启动
/usr/local/nginx/sbin/nginx
ss -lnt | grep 80

5：部署后端Nginx服务器
nginx_server_1
nginx_server_2
# 在这两台机器操作

yum install -y nginx
# 输出识别表示
echo "<h1>Server-1</h1>" >/usr/share/nginx/html/index.html (nginx_server_1)
echo "<h1>Server-2</h1>" >/usr/share/nginx/html/index.html (nginx_server_2)
nginx (启动Nginx服务器)
ss -lnt | grep 80 (检测服务器是否正常启动)

3：测试

1：测试负载均衡
C:\Users\Administrator>curl 10.0.0.100
<h1>Server-2</h1>
C:\Users\Administrator>curl 10.0.0.100
<h1>Server-1</h1>

[root@keepalived-tengine-master ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:96:1e:ca brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/8 brd 10.255.255.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::fb52:3261:d026:fd4a/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

2：down掉负载master负载均衡
/usr/local/nginx/sbin/nginx -s stop
[root@keepalived-tengine-master ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:96:1e:ca brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/8 brd 10.255.255.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::fb52:3261:d026:fd4a/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever


3：查看BACKUP
[root@keepalived-tengine-backup ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:07:2a:90 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.11/8 brd 10.255.255.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::fb52:3261:d026:fd4a/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::3dfc:7872:113d:c769/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::ecc5:6970:d857:1f9f/64 scope link tentative noprefixroute dadfailed 
       valid_lft forever preferred_lft forever

4：再次测试
C:\Users\Administrator>curl 10.0.0.100
<h1>Server-1</h1>

C:\Users\Administrator>curl 10.0.0.100
<h1>Server-2</h1>

结束