Apache+Tomcat构建Tomcat负载均衡集群
一、环境介绍
二、安装后端服务器
三、安装前端Apache服务
四、配置Apache使用mod_jk模块实现代理及负载均衡
五、配置Apache基于mod_proxy模块实现代理及负载均衡
六、论坛安装
七、安装并配置Keepalived实现前端高可用
一、环境介绍
系统版本:CentOS 6.4_x86_64
Mysql版本:mysql-5.1.66-2.el6_3.x86_64
Apache版本:httpd-2.2.25 点此下载
Keepalived版本:keepalived-1.2.7 点此下载
Tomcat版本:apache-tomcat-7.0.33 点此下载
Jdk版本: jdk-7u40-linux-x64 点此下载
Tomcat-commectors版本:tomcat-connectors-1.2.37 点此下载
论坛版本:JspRun!_6.0.0_GBK 点此下载
环境介绍:
当用户通过 "www.allen.com" 域名访问时,首先是由前端两台Apache代理服务器响应并转发到后端Tomcat服务器上,而在Apache上做的是反向代理负载均衡到后端Tomcat服务器上,前端服务器使用Keepalived做的高可用集群,【Apache1默认为Master,Apache2为Backup】虚拟IP地址为"172.16.14.10"模拟为公网IP;当然这里数据库只有一台主机会成为点故障,这里不要介绍Mysql的高可用,如果有兴趣可以看前面写的博客
提示:这里使用Keepalived做高可用时,前面所有的服务都已完成后,最后实现的对前端服务器做高可用
二、安装后端Tomcat与Mysql服务器
1、在Tomcat1与Tomcat2服务器上安装Tomcat,安装方法相同,这里只介绍一次
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
|
安装JDK # rpm -ivh jdk-7u40-linux-x64.rpm # vim /etc/profile.d/java.sh export JAVA_HOME= /usr/java/latest export PATH=$JAVA_HOME /bin :$PATH # . /etc/profile.d/java.sh # java -version java version "1.7.0_40" Java(TM) SE Runtime Environment (build 1.7.0_40-b43) Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode) ------------------------------------------------------------------------ 安装Tomcat # tar xf apache-tomcat-7.0.33.tar.gz -C /usr/local/ # cd /usr/local/ # ln -s apache-tomcat-7.0.33 tomcat # vim /etc/profile.d/tomcat.sh export CATALINA_HOME= /usr/local/tomcat export PATH=$CATALINA_HOME /bin :$PATH # . /etc/profile.d/tomcat.sh # catalina.sh version Using CATALINA_BASE: /usr/local/tomcat Using CATALINA_HOME: /usr/local/tomcat Using CATALINA_TMPDIR: /usr/local/tomcat/temp Using JRE_HOME: /usr/java/latest Using CLASSPATH: /usr/local/tomcat/bin/bootstrap .jar: /usr/local/tomcat/bin/tomcat-juli .jar Server version: Apache Tomcat /7 .0.33 Server built: Nov 18 2012 04:15:21 Server number: 7.0.33.0 OS Name: Linux OS Version: 2.6.32-358.el6.x86_64 Architecture: amd64 JVM Version: 1.7.0_40-b43 JVM Vendor: Oracle Corporation |
2、启动Tomcat服务并访问测试,默认访问端口为"8080";如下
1
2
3
4
5
6
7
8
9
10
11
|
# catalina.sh start Using CATALINA_BASE: /usr/local/tomcat Using CATALINA_HOME: /usr/local/tomcat Using CATALINA_TMPDIR: /usr/local/tomcat/temp Using JRE_HOME: /usr/java/latest Using CLASSPATH: /usr/local/tomcat/bin/bootstrap .jar: /usr/local/tomcat/bin/tomcat-juli .jar # jps 2283 Jps 2272 Bootstrap # ss -tanlp | grep 8080 LISTEN 0 100 :::8080 :::* users :(( "java" ,2272,40)) |
3、开启Tomcat管理及状态页面
1
2
3
4
5
6
|
# vim /usr/local/tomcat/conf/tomcat-users.xml <role rolename= "manager-gui" /> <role rolename= "admin-gui" /> <user username= "allen" password= "admin" roles= "manager-gui,admin-gui" /> # catalina.sh stop # catalina.sh start |
4、访问Tomcat后台管理
5、安装Mysql服务器
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
######安装Mysql并创建数据库,授权用户访问新创建的数据库有所有权,方便后面使用 [root@Mysql ~] # yum -y install mysql-server [root@Mysql ~] # service mysqld start [root@Mysql ~] # mysql mysql> create database bbs; mysql> grant all on bbs.* to 'bbsuser' @ '172.16.%.%' identified by 'bbspass' ; mysql> flush privileges; ---------------------------------------------------------------------- 远程连接数据库测试授权用户是否能连接成功 # mysql -ubbsuser -pbbspass -h 172.16.14.5 -e 'show databases'; +--------------------+ | Database | +--------------------+ | information_schema | | bbs | | test | +--------------------+ |
三、安装前端Apache服务
1、在Apache1与Apache2服务器上安装Httpd软件,安装方法相同,这里只介绍一次
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
# tar xf httpd-2.2.25.tar.bz2 # cd httpd-2.2.25 # yum -y install gcc gcc-c++ openssl-devel pcre-devel # ./configure --prefix=/usr/local/apache --sysconfdir=/etc/httpd --enable-modules=most --enable-so --enable-ssl --enable-cgi --enable-rewrite --with-zlib --with-pcre --enable-mpms-shared=all --with-mpm=prefork --enable-proxy --enable-proxy-http --enable-proxy-ajp --enable-proxy-balancer --enable-cache --enable-cache-disk --enable-mem-cache --enable-file-cache --enable-proxy-connect # make && make install # cp build/rpm/httpd.init /etc/init.d/httpd ------修改如下内容 # vim /etc/init.d/httpd httpd=${HTTPD- /usr/local/apache/bin/httpd } pidfile=${PIDFILE- /usr/local/apache/logs/ ${prog}.pid} lockfile=${LOCKFILE- /var/lock/subsys/ ${prog}} RETVAL=0 # check for 1.3 configuration check13 () { CONFFILE= /etc/httpd/httpd .conf ---------------------------------------------------------------------- 修改环境变量并把Httpd加入到系统服务 # echo "PATH=/usr/local/apache/bin:$PATH" >> /etc/profile.d/http.sh # . /etc/profile.d/http.sh # ln -s /usr/local/apache/include/ /usr/include/httpd # chkconfig --add httpd ---------------------------------------------------------------------- 创建测试页 # echo "<h1>APACHE</h1>" > /usr/local/apache/htdocs/index.html |
2、在Apache1与Apache2服务器上安装"tomcat-connectors"
1
2
3
4
5
6
7
8
9
10
11
12
13
|
# tar xf tomcat-connectors-1.2.37-src.tar.gz # cd tomcat-connectors-1.2.37-src/native/ # ./configure --with-apxs=/usr/local/apache/bin/apxs # make && make install ------------------------------------------------------------------------ 安装完成后会生成如下文件 # ls /usr/local/apache/modules/ mod_jk.so ------------------------------------------------------------------------ 启动Httpd服务 #service httpd start # ss -tanl |grep 80 LISTEN 0 128 :::80 :::* |
3、访问测试Apache是否正常工作
四、配置Apache使用mod_jk模块实现代理及负载均衡
1、修改Apache主配置文件,包含一个文件并创建该文件;实现代理功能
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
将下面指令添加到文件末尾即可 # vim /etc/httpd/httpd.conf Include /etc/httpd/extra/mod_jk .conf ------------------------------------------------------------------------ 创建文件添加如下内容 # vim /etc/httpd/extra/mod_jk.conf LoadModule jk_module modules /mod_jk .so JkWorkersFile /etc/httpd/extra/workers .properties JkLogFile logs /mod_jk .log JkLogLevel debug JkMount /* TomcatA JkMount /status/ stat1 ------------------------------------------------------------------------ # vim /etc/httpd/extra/workers.properties worker.list=TomcatA,stat1 worker.TomcatA.port=8009 worker.TomcatA.host=172.16.14.3 worker.TomcatA. type =ajp13 worker.TomcatA.lbfactor=1 worker.stat1. type = status ---------------------------------------------------------------------- # service httpd restart |
2、修改后端Tomcat1服务器,添加一个虚拟主机并提供测试页面
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
|
[root@Tomcat1 ~] # cd /usr/local/tomcat/conf/ [root@Tomcat1 conf] # cp server.xml server.xml.bak [root@Tomcat1 conf] # vim server.xml ######修改如下内容 <Engine name= "Catalina" defaultHost= "www.allen.com" jvmRoute= "TomcatA" > ######在"Engine"中添加如下内容 <Host name= "www.allen.com" appBase= "/web/allen" unpackWARs= "true" autoDeploy= "true" > <Valve className= "org.apache.catalina.valves.AccessLogValve" directory= "logs" prefix= "allen_access_log." suffix= ".txt" pattern= "%h %l %u %t "%r" %s %b" /> <Context path= "" docBase= "/web/allen" /> < /Host > ---------------------------------------------------------------------- ######创建网站存放目录并创建测试页 [root@Tomcat1 ~] # mkdir -p /web/allen [root@Tomcat1 ~] # vim /web/allen/index.jsp <%@ page language= "java" %> <html> < head ><title>TomcatA< /title >< /head > <body> <h1><font color= "red" >TomcatA < /font >< /h1 > <table align= "centre" border= "1" > < tr > <td>Session ID< /td > <% session.setAttribute( "abc" , "abc" ); %> <td><%= session.getId() %>< /td > < /tr > < tr > <td>Created on< /td > <td><%= session.getCreationTime() %>< /td > < /tr > < /table > < /body > < /html > ---------------------------------------------------------------------- ######重启Tomcat服务并查看是否启动成功 [root@Tomcat1 ~] # catalina.sh stop [root@Tomcat1 ~] # catalina.sh configtest [root@Tomcat1 ~] # catalina.sh start [root@Tomcat1 ~] # ss -tanlp | grep java LISTEN 0 100 :::8080 :::* users :(( "java" ,6323,40)) LISTEN 0 1 ::ffff:127.0.0.1:8005 :::* users :(( "java" ,6323,52)) LISTEN 0 100 :::8009 :::* users :(( "java" ,6323,41)) |
3、访问Apache1主机,验证是否代理成功
4、修改Apache配置文件(mod_jk.conf、workers.properties);实现负载均衡
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
# vim /etc/httpd/extra/mod_jk.conf LoadModule jk_module modules /mod_jk .so JkWorkersFile /etc/httpd/extra/workers .properties JkLogFile logs /mod_jk .log JkLogLevel debug JkMount /* lbcluster JkMount /status/ stat1 ------------------------------------------------------------------------ # vim /etc/httpd/extra/workers.properties worker.list=lbcluster,stat1 worker.TomcatA.port=8009 worker.TomcatA.host=172.16.14.3 worker.TomcatA. type =ajp13 worker.TomcatA.lbfactor=1 worker.TomcatB.port = 8009 worker.TomcatB.host=172.16.14.4 worker.TomcatB. type = ajp13 worker.TomcatB.lbfactor = 1 worker.lbcluster. type = lb worker.lbcluster.sticky_session = 0 #取值为{1|0}1将用户session与后端服务器绑定,0为不绑定,如果支持session复制或session共享可以设置为0 worker.lbcluster.balance_workers = TomcatA, TomcatB worker.stat1. type = status ---------------------------------------------------------------------- # service httpd reload |
5、修改后端Tomcat2服务器,添加一个虚拟主机并提供测试页面
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
|
[root@Tomcat2 ~] # cd /usr/local/tomcat/conf/ [root@Tomcat2 conf] # cp server.xml server.xml.bak [root@Tomcat2 conf] # vim server.xml ######修改如下内容 <Engine name= "Catalina" defaultHost= "www.allen.com" jvmRoute= "TomcatB" > ######在"Engine"中添加如下内容 <Host name= "www.allen.com" appBase= "/web/allen" unpackWARs= "true" autoDeploy= "true" > <Valve className= "org.apache.catalina.valves.AccessLogValve" directory= "logs" prefix= "allen_access_log." suffix= ".txt" pattern= "%h %l %u %t "%r" %s %b" /> <Context path= "" docBase= "/web/allen" /> < /Host > ---------------------------------------------------------------------- ######创建网站存放目录并创建测试页 [root@Tomcat2 ~] # mkdir -p /web/allen [root@Tomcat2 ~] # vim /web/allen/index.jsp <%@ page language= "java" %> <html> < head ><title>TomcatB< /title >< /head > <body> <h1><font color= "blue" >TomcatB < /font >< /h1 > <table align= "centre" border= "1" > < tr > <td>Session ID< /td > <% session.setAttribute( "abc" , "abc" ); %> <td><%= session.getId() %>< /td > < /tr > < tr > <td>Created on< /td > <td><%= session.getCreationTime() %>< /td > < /tr > < /table > < /body > < /html > ---------------------------------------------------------------------- ######重启Tomcat服务并查看是否启动成功 [root@Tomcat2 ~] # catalina.sh stop [root@Tomcat2 ~] # catalina.sh configtest [root@Tomcat2 ~] # catalina.sh start [root@Tomcat2 ~] # ss -tanlp | grep java LISTEN 0 100 :::8080 :::* users :(( "java" ,6323,40)) LISTEN 0 1 ::ffff:127.0.0.1:8005 :::* users :(( "java" ,6323,52)) LISTEN 0 100 :::8009 :::* users :(( "java" ,6323,41)) |
6、访问Apache服务器,测试是否实现负载均衡
由上可见,基于Apache的mod_jk模块已成功实现负载均衡功能
五、配置Apache基于mod_proxy模块实现代理及负载均衡
1、修改Apache的主配置文件,包含一个文件并创建该文件;实现代理功能
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
# vim /etc/httpd/httpd.conf #Include /etc/httpd/extra/mod_jk.conf #注释此行 Include /etc/httpd/extra/mod_proxy .conf ------------------------------------------------------------------------ # vim /etc/httpd/extra/mod_proxy.conf ProxyVia On ProxyRequests Off ProxyPreserveHost Off <Proxy *> Order allow,deny Allow from all < /Proxy > ProxyPass / ajp: //172 .16.14.3:8009/ ProxyPassReverse / ajp: //172 .16.14.3:8009/ <Location / > Order allow,deny Allow from all < /Location > ---------------------------------------------------------------------- # service httpd reload ====================================================================== 注释: ProxyPass / ajp: //172 .16.14.3:8009/ #使用的是ajp协议 ProxyPassReverse / ajp: //172 .16.14.3:8009/ 可以更改为使用http协议;如下 ProxyPass / http: //172 .16.14.3:8080/ ProxyPassReverse / http: //172 .16.14.3:8080/ ######这里只演示使用ajp协议,如果是Apache与Tomcat结合建议使用ajp协议 |
2、访问Apache1服务器,测试是否代理成功
3、修改Apache配置文件(mod_proxy.conf);实现负载均衡
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
|
# vim /etc/httpd/extra/mod_proxy.conf ProxyVia Off ProxyRequests Off ProxyPreserveHost Off <Proxy balancer: //allen > BalancerMember ajp: //172 .16.14.3:8009 loadfactor=1 BalancerMember ajp: //172 .16.14.4:8009 loadfactor=1 ProxySet lbmethod=bytraffic < /Proxy > <Location /allen > #设置状态及管理页面 SetHandler balancer-manager Proxypass ! Order allow,deny Allow from all < /Location > <Proxy *> Order allow,deny Allow from all < /Proxy > ProxyPass / balancer: //allen/ stickysession=JSESSIONID ProxyPassReverse / balancer: //allen/ <Location / > Order allow,deny Allow from all < /Location > ---------------------------------------------------------------------- 注释: BalancerMember ajp: //172 .16.14.3:8009 loadfactor=1 #使用ajp协议 BalancerMember ajp: //172 .16.14.4:8009 loadfactor=1 可更改为使用http协议 BalancerMember http: //172 .16.14.3:8080 loadfactor=1 BalancerMember http: //172 .16.14.4:8080 loadfactor=1 ######这里只介绍使用ajp协议,如果有兴趣可以更改一下;loadfactor:权重 ---------------------------------------------------------------------- ProxySet lbmethod=bytraffic #设置调度算法 byrequests:即基于权重将统计请求个数进行调度(默认) bytraffic:则执行基于权重的流量计数调度 bybusyness:通过考量每个后端服务器的当前负载进行调度 ---------------------------------------------------------------------- # service httpd reload |
4、访问Apache服务器,测试基于mod_proxy模块是否实现负载均衡
5、访问状态页面
六、论坛安装
1、在Tomcat1服务器上安装
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
# unzip JspRun\!_6.0.0_GBK.zip # mv /web/allen/index.jsp /web/allen/test.jsp # cp -r upload/* /web/allen/ ------------------------------------------------------------------------ 修改论坛数据库连接文件 # vim /web/allen/config.properties dbhost = 172.16.14.5 dbport=3306 dbuser = bbsuser dbpw = bbspass dbname = bbs ------------------------------------------------------------------------ # catalina.sh stop # catalina.sh start |
2、安装论坛程序
3、点击下一步-->我同意-->下一步,进入数据库配置页面
4、点下一步-->(填写管理员密码)下一步-->进入创建数据表阶段-->然后点-->进入首页
5、将论坛程序拷贝到Tomcat2服务器上一份并访问测试
1
2
3
4
5
|
# mv /web/allen/index.jsp /web/allen/test.jsp #把原来的测试页重命名 ------------------------------------------------------------------------ # scp -r 172.16.14.3:/web/allen/* /web/allen/ # catalina.sh stop # catalina.sh start |
6、访问Apache服务器,测试是否能正常访问论坛程序,这里就不在做访问测试
七、安装并配置Keepalived
1、将Apache1服务器的Httpd配置文件复制到Apache2服务器
1
2
3
4
|
[root@Apache1 ~] # cd /etc/httpd/ [root@Apache1 httpd] # scp httpd.conf 172.16.14.2:/etc/httpd/ [root@Apache1 httpd] # cd extra/ [root@Apache1 extra] # scp mod_jk.conf mod_proxy.conf workers.properties 172.16.14.2:/etc/httpd/extra/ |
2、在Apache1与Apache2服务器上安装Keepalived;这里使用rpm包安装,光盘映像中有
1
2
3
|
[root@Apache1 ~] # yum -y install keepalived -------------------------------------------- [root@Apache2 ~] # yum -y install keepalived |
3、配置Apache1服务器上的Keepalived
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
|
[root@Apache1 ~] # vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from admin@allen.com smtp_server 172.16.0.0 smtp_connect_timeout 30 router_id LVS_ALLEN } vrrp_script chk_httpd { script "killall -0 httpd" interval 1 weight -2 } vrrp_instance httpd_1 { state MASTER interface eth0 virtual_router_id 58 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1058 } virtual_ipaddress { 172.16.14.10 } track_script { chk_httpd } } [root@Apache1 ~] # service keepalived start [root@Apache1 ~] # chkconfig keepalived on [root@Apache1 ~] # chkconfig --list keepalived keepalived 0:off 1:off 2:on 3:on 4:on 5:on 6:off [root@Apache1 ~] # ip addr show eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link /ether 00:0c:29:2c:1a:24 brd ff:ff:ff:ff:ff:ff inet 172.16.14.1 /16 brd 172.16.255.255 scope global eth0 inet 172.16.14.10 /32 scope global eth0 inet6 fe80::20c:29ff:fe2c:1a24 /64 scope link valid_lft forever preferred_lft forever |
4、配置Apache2服务器上的Keepalived
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
|
[root@Apache2 ~] # vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from admin@allen.com smtp_server 172.16.0.0 smtp_connect_timeout 30 router_id LVS_ALLEN } vrrp_script chk_httpd { script "killall -0 httpd" interval 1 weight -2 } vrrp_instance httpd_1 { state BACKUP interface eth0 virtual_router_id 58 priority 99 advert_int 1 authentication { auth_type PASS auth_pass 1058 } virtual_ipaddress { 172.16.14.10 } track_script { chk_httpd } } [root@Apache2 ~] # service keepalived start [root@Apache2 ~] # chkconfig --list keepalived keepalived 0:off 1:off 2:on 3:on 4:on 5:on 6:off |
5、使用"www.allen.com" 域名访问,验证是否能访问到论坛;由于没有DNS服务器解析,修改了"hosts"文件;如:
1
2
3
|
######添加如下内容解析 C:\Windows\System32\drivers\etc\hosts 172.16.14.10 www.allen.com |
6、模拟前端一台服务器出现故障;查看虚拟IP是否转移
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
######停止Apache1服务器上的Httpd服务来模拟故障;并查看IP [root@Apache1 ~] # service httpd stop [root@Apache1 ~] # ip addr show eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link /ether 00:0c:29:2c:1a:24 brd ff:ff:ff:ff:ff:ff inet 172.16.14.1 /16 brd 172.16.255.255 scope global eth0 inet6 fe80::20c:29ff:fe2c:1a24 /64 scope link valid_lft forever preferred_lft forever ======================================================================== ######查看Apache2服务器上的IP [root@Apache2 ~] # ip addr show eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link /ether 00:0c:29:ec:f6:3f brd ff:ff:ff:ff:ff:ff inet 172.16.14.2 /16 brd 172.16.255.255 scope global eth0 inet 172.16.14.10 /32 scope global eth0 inet6 fe80::20c:29ff:feec:f63f /64 scope link valid_lft forever preferred_lft forever |
7、从上一步可以看出,虚拟IP地址已成功转移,说明还能正常提供服务,可以再次访问测试;如果服务器修复好重新上线,虚拟IP还会转移回去;这里就不再测试
到此,基于Apache做反向代理实现了Tomcat的负载均衡;而Keepalived实现了前端服务器的高可用;关于session保存的问题,方案有很多,比如可以保存到"memcached"、"redis"等,使用哪种方案具体还要根据自己的需求而定...