FreeBSD 7.0下通过CARP+PF+PFSYNC实现服务器群集笔记

 关于carp+pf+pfsync的群集, 网上大多是OB的资料, 于是找到些机器将其移到FB上来. 实验上是成功,可能存在一些问题, 请大家来发现和指正!!!!
硬件:
服务器                5台(实际只用了4台, 2台做LoadBalance(主/备各一,每台服务器至少需要三块网卡), 2台做应用服务.)
cisco2950        2台
系统: FreeBSD 7.0 release(最小化安装)
实现目标:
1. 通过pfsync, 实现LoadBalance 的 高可用性(HA);
2. 实现服务器池的负载均衡;
我部署环境的测试地址(不能长期保留, 群集部署是在内网中,所以外网通过重定向来carp0上的):
测试地址就不放上来了
一 网络拓扑:
外网: 192.168.1.0/24
内网: 192.168.10.0/24
HA专用: 10.10.10.250 / 10.10.10.251
screen.width*0.7) {this.resized=true; this.width=screen.width*0.7; this.style.cursor='hand'; this.alt='Click here to open new window\nCTRL+Mouse wheel to zoom in/out';}" onclick="if(!this.resized) {return true;} else {window.open('http://bbs.chinaunix.net/attachments/month_0805/20080515_659d2f4c6701ca7fd002d4DihrUqt2c3.png');}" alt="" src="http://bbs.chinaunix.net/attachments/month_0805/20080515_659d2f4c6701ca7fd002d4DihrUqt2c3.png" onload="if(this.width>screen.width*0.7) {this.resized=true; this.width=screen.width*0.7; this.alt='Click here to open new window\nCTRL+Mouse wheel to zoom in/out';}" border=0>
screen.width*0.7) {this.resized=true; this.width=screen.width*0.7; this.style.cursor='hand'; this.alt='Click here to open new window\nCTRL+Mouse wheel to zoom in/out';}" onclick="if(!this.resized) {return true;} else {window.open('http://bbs.chinaunix.net/images/default/attachimg.gif');}" alt="" src="http://bbs.chinaunix.net/images/default/attachimg.gif" onload="if(this.width>screen.width*0.7) {this.resized=true; this.width=screen.width*0.7; this.alt='Click here to open new window\nCTRL+Mouse wheel to zoom in/out';}" border=0>
二 Load Balance配置:
A.内核设置:
master/slave的内核配置是一样的
device carp
device pf        #启动虚拟网络设备来记录流量(经由 bpf)
device pflog     #启动虚拟网络设备来监视网络状态
device pfsync
options         ALTQ
options         ALTQ_CBQ        # 基于分类的排列 (CBQ)
options         ALTQ_RED        # 随机先期检测 (RED)
options         ALTQ_RIO        # 对进入和发出的包进行 RED
options         ALTQ_HFSC       # 带等级的包调度器 (HFSC)
options         ALTQ_PRIQ       # 按优先级的排列 (PRIQ)
options         ALTQ_NOPCC      # 在联编 SMP 内核时必须使用，禁止读时钟
重编内核.
B.配置rc.conf
这里只写与本章节相关的配置项了.
master部分:
gateway_enable="YES"
defaultrouter="192.168.1.1"
hostname="master.cluster.org"
cloned_interfaces="carp0 carp1"
# External Public Interface (for the secondary firewall use a different public ip.)
ifconfig_em0="inet 192.168.1.52 netmask 255.255.255.0"
# External Public Carp Interface
#ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24"
ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24 advskew 10"
# Internal Interface (for the secondary firewall change the ip address to 192.168.10.11)
ifconfig_em1="inet 192.168.10.10 netmask 255.255.255.0"
# Internal Carp Interface
ifconfig_carp1="vhid 1 pass 22222 192.168.10.100/24 advskew 10"
# Heartbeat Interface (for the secondary firewall, change the ip address to 10.10.10.251)
ifconfig_vr0="10.10.10.250 netmask 255.255.255.0"
# PFSync Interface
ifconfig_pfsync0="up syncif vr0"
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""
slave部分:
#slave的配置与master配置大概相近了. 改动的主要是本机IP和advskew的优先值. 注意: carp 的IP是公共的必须一致.
gateway_enable="YES"
defaultrouter="192.168.1.1"
hostname="slave.cluster.org"
cloned_interfaces="carp0 carp1"
# External Public Interface (for the primary firewall use a different public ip.)
ifconfig_em0="inet 192.168.1.53 netmask 255.255.255.0"
# External Public Carp Interface
#ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24"
ifconfig_carp0="vhid 1 pass 11111 192.168.1.51/24 advskew 20"
# Internal Interface (for the primary firewall change the ip address to 192.168.10.10)
ifconfig_em1="inet 192.168.10.11 netmask 255.255.255.0"
# Internal Carp Interface
ifconfig_carp1="vhid 1 pass 22222 192.168.10.100/24 advskew 20"
# Heartbeat Interface (for the primary firewall, change the ip address to 10.10.10.250)
ifconfig_fxp0="10.10.10.251 netmask 255.255.255.0"
# PFSync Interface
ifconfig_pfsync0="up syncif fxp0"
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""
C.pf.conf规则
master和slave除了网卡标识不一样.其他是一致的.
################################################################################
# Macro and lists
################################################################################
lop_if = "lo0"
ext_if = "em0"
int_if = "em1"
sync_if= "vr0"
ext_carp = "carp0"
web_ports = "{ 80, 443 }"
#web_servers = "{ 192.168.10.20, 192.168.10.21, 192.168.10.22 }"
#web_servers = "{ 192.168.10.20 }"
web_servers = "{ 192.168.10.20, 192.168.10.21 }"
################################################################################
# Options, scrub and NAT
################################################################################
set block-policy drop
set skip on $lop_if
scrub in
nat on $ext_if from $int_if:network to any -> $ext_if
################################################################################
# Redirection
################################################################################
#rdr on $ext_if proto tcp from any to any port 80 -> $web_servers round-robin sticky-address
rdr on $ext_if proto tcp from any to $ext_carp port $web_ports -> $web_servers round-robin sticky-address
################################################################################
# Filtering Rules
################################################################################
pass quick on { $sync_if } proto pfsync keep state (no-sync)
pass on { $ext_if,$int_if } proto carp keep state
D. sysctl.conf的相关配置
#master和slave 一样.
net.inet.carp.preempt=1
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536
E. 配置好后重启服务器.检查系统状况
master上
master# ifconfig
vr0: flags=8843 metric 0 mtu 1500
        options=8
        ether 00:05:5d:85:84:d8
        inet 10.10.10.250 netmask 0xffffff00 broadcast 10.10.10.255
        media: Ethernet autoselect (100baseTX )
        status: active
em0: flags=8943 metric 0 mtu 1500
        options=9b
        ether 00:c0:9f:31:25:a2
        inet 192.168.1.52 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (100baseTX )
        status: active
em1: flags=8943 metric 0 mtu 1500
        options=9b
        ether 00:c0:9f:31:25:a3
        inet 192.168.10.10 netmask 0xffffff00 broadcast 192.168.10.255
        media: Ethernet autoselect (100baseTX )
        status: active
pflog0: flags=141 metric 0 mtu 33204
lo0: flags=8049 metric 0 mtu 16384
        inet 127.0.0.1 netmask 0xff000000
pfsync0: flags=41 metric 0 mtu 1460
        pfsync: syncdev: vr0 syncpeer: 224.0.0.240 maxupd: 128
carp0: flags=49 metric 0 mtu 1500
        inet 192.168.1.51 netmask 0xffffff00
        carp: MASTER vhid 1 advbase 1 advskew 10
carp1: flags=49 metric 0 mtu 1500
        inet 192.168.10.100 netmask 0xffffff00
        carp: MASTER vhid 1 advbase 1 advskew 10
slave上:  
slave# ifconfig
fxp0: flags=8843 metric 0 mtu 1500
        options=8
        ether 00:07:e9:1b:4b:cd
        inet 10.10.10.251 netmask 0xffffff00 broadcast 10.10.10.255
        media: Ethernet autoselect (100baseTX )
        status: active
em0: flags=8943 metric 0 mtu 1500
        options=9b
        ether 00:c0:9f:38:bd:af
        inet 192.168.1.53 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (100baseTX )
        status: active
em1: flags=8943 metric 0 mtu 1500
        options=9b
        ether 00:c0:9f:38:bd:b0
        inet 192.168.10.11 netmask 0xffffff00 broadcast 192.168.10.255
        media: Ethernet autoselect (100baseTX )
        status: active
lo0: flags=8049 metric 0 mtu 16384
        inet 127.0.0.1 netmask 0xff000000
pflog0: flags=141 metric 0 mtu 33204
pfsync0: flags=41 metric 0 mtu 1460
        pfsync: syncdev: fxp0 syncpeer: 224.0.0.240 maxupd: 128
carp0: flags=49 metric 0 mtu 1500
        inet 192.168.1.51 netmask 0xffffff00
        carp: BACKUP vhid 1 advbase 1 advskew 20
carp1: flags=49 metric 0 mtu 1500
        inet 192.168.10.100 netmask 0xffffff00
        carp: BACKUP vhid 1 advbase 1 advskew 20
二. 服务器池的配置.
统一配置,把网关指向192.168.10.100就可以了.
defaultrouter="192.168.10.100"
我在二台服务器上部署了相同的应用(lighttpd + php-fcgi), 配置我就不写了. 大家GG一下.
由于服务器不够,mysql我部署在其中的s2上.让两台服务器连到他上.
三 测试.
A.静态页面:
在s1和s2上分别与建立个index.htm页面.
s1上index.htm的内容是:
"hi, this is No.1 server"
s2上index.htm的内容是:
"hi, this is No.2 server"
访问地址:  
http://192.168.1.51/index.htm
返回的内容分别s1和s2的. 注意: 调度上做的状态保持的. 所以需要关闭浏览器再开才分别看得到不同的内容. 建议最好是几个人一起访问来返回不同的结果.
B. 动态页面:
在s1和s2上简单部署了phpwind.  假如在./bbs下.
访问地址:
http://192.168.1.51/bbs/
返回的内容保持了一致性,
这里要提到的是文件的共享我没做, 现实的方法可以是通过nfs和iscsi. 暂时调不出iscsi的存储, 又不想用nfs所以就不作了. :)
master# pfctl -s state
all carp 192.168.1.52 -> 224.0.0.18       SINGLE:NO_TRAFFIC
all pfsync 224.0.0.240  224.0.0.240       SINGLE:NO_TRAFFIC
all carp 224.0.0.18  224.0.0.18       SINGLE:NO_TRAFFIC
all tcp 192.168.10.21:65346 -> 192.168.1.52:61575 -> 130.104.5.67:25  FIN_WAIT_2:FIN_WAIT_2
all tcp 192.168.10.20:80  metric 0 mtu 1500
        inet 192.168.1.51 netmask 0xffffff00
        carp: MASTER vhid 1 advbase 1 advskew 20
大家可以做些些测试了.
最后能有些优化:)
文献参考:
http://www.countersiege.com/doc/pfsync-carp/
http://blog.randomutterings.com/articles/2007/06/15/redundant-failover-firewall-with-pf-pfsync-and-carp-on-freebsd