Centos下Apache+Tomcat集群--搭建记录
一.目的
利用apache的mod_jk模块，实现tomcat集群服务器的负载均衡以及会话复制，这里用到了<Cluster>.
二.环境
1.基础：3台主机，系统Centos6.5，4G内存，50G硬盘. yum源已更换为阿里源（如何更换可参考博客的另外一篇文章CDN）
2.软件：development tools，jdk-7u9-linux-x64.rpm;源码编译软件apr-util-1.5.4.tar.gz,apr-1.5.2.tar.gz,tomcat-connectors-1.2.40-src.tar.gz,httpd-2.4.16.tar.gz
3.源码包下载地址:可以去官网下载，或者在此网盘内下载：点击链接
4.拓扑图
5.现在就可以安装部署了。
三.部署安装
Apache环节：
1.编译安装apr
1 [root@apache down]# tar xf apr-1.5.2.tar.gz 2 [root@apache down]# cd apr-1.5.2 3 [root@apache apr-1.5.2]# ./configure --prefix=/usr/local/apr --disable-ipv6 4 [root@apache apr-1.5.2]# make && make install
2.编译安装apr-util
1 [root@apache down]# tar xf apr-util-1.5.4.tar.gz 2 [root@apache down]# cd apr-util-1.5.4 3 [root@apache apr-util-1.5.4]# ./configure --prefix=/usr/local/apr-util --with-apr=/usr/local/apr 4 [root@apache apr-util-1.5.4]# make && make install
3.编译安装httpd
1 [root@apache ~]# yum -y install pcre-devel openssl-devel (编译安装前，先安装这两个软件，解决编译httpd时的依赖问题)
解压httpd，并编译安装。（如果在编译出错情况，请仔细排查错误，记得重新编译时，需要make clean 清除一下）
1 [root@apache down]# tar xf httpd-2.4.16.tar.gz 2 [root@apache down]# cd httpd-2.4.16 3 [root@apache httpd-2.4.16]# ./configure --prefix=/usr/local/apache --sysconfdir=/etc/httpd --enable-so --enable-ssl --enable-cgi --enable-rewrite --with-zlib --with-pcre --with-apr=/usr/local/apr --with-apr-util=/usr/local/apr-util --enable-mpms-shared=all --with-mpm=event --enable-proxy --enable-proxy-http --enable-proxy-ajp --enable-proxy-balancer --enable-lbmethod-heartbeat --enable-heartbeat --enable-slotmem-shm --enable-slotmem-plain --enable-watchdog 4 [root@apache httpd-2.4.16]# make && make install
为apache 提供httpd脚本，并加入系统服务，设为开机启动，方便以后的服务管理,文件路径 /etc/rc.d/init.d/httpd
[root@apache httpd-2.4.16]# vim /etc/rc.d/init.d/httpd #!/bin/bash # # httpd Startup script for the Apache HTTP Server # # chkconfig: - 85 15 # description: Apache is a World Wide Web server. It is used to serve \ # HTML files and CGI. # processname: httpd # config: /etc/httpd/conf/httpd.conf # config: /etc/sysconfig/httpd # pidfile: /var/run/httpd.pid # Source function library. . /etc/rc.d/init.d/functions if [ -f /etc/sysconfig/httpd ]; then . /etc/sysconfig/httpd fi # Start httpd in the C locale by default. HTTPD_LANG=${HTTPD_LANG-"C"} # This will prevent initlog from swallowing up a pass-phrase prompt if # mod_ssl needs a pass-phrase from the user. INITLOG_ARGS="" # Set HTTPD=/usr/sbin/httpd.worker in /etc/sysconfig/httpd to use a server # with the thread-based "worker" MPM; BE WARNED that some modules may not # work correctly with a thread-based MPM; notably PHP will refuse to start. # Path to the apachectl script, server binary, and short-form for messages. apachectl=/usr/local/apache/bin/apachectl httpd=${HTTPD-/usr/local/apache/bin/httpd} prog=httpd pidfile=${PIDFILE-/var/run/httpd.pid} lockfile=${LOCKFILE-/var/lock/subsys/httpd} RETVAL=0 start() { echo -n $"Starting $prog: " LANG=$HTTPD_LANG daemon --pidfile=${pidfile} $httpd $OPTIONS RETVAL=$? echo [ $RETVAL = 0 ] && touch ${lockfile} return $RETVAL } stop() { echo -n $"Stopping $prog: " killproc -p ${pidfile} -d 10 $httpd RETVAL=$? echo [ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile} } reload() { echo -n $"Reloading $prog: " if ! LANG=$HTTPD_LANG $httpd $OPTIONS -t >&/dev/null; then RETVAL=$? echo $"not reloading due to configuration syntax error" failure $"not reloading $httpd due to configuration syntax error" else killproc -p ${pidfile} $httpd -HUP RETVAL=$? fi echo } # See how we were called. case "$1" in start) start ;; stop) stop ;; status) status -p ${pidfile} $httpd RETVAL=$? ;; restart) ;; condrestart) if [ -f ${pidfile} ] ; then stop start fi ;; reload) reload ;; graceful|help|configtest|fullstatus) $apachectl $@ RETVAL=$? ;; *) echo $"Usage: $prog {start|stop|restart|condrestart|reload|status|fullstatus|graceful|help|configtest}" exit 1 esac exit $RETVAL
脚本编辑完成后，添加为系统服务
1 [root@apache httpd-2.4.16]# chkconfig --add httpd 2 [root@apache httpd-2.4.16]# chkconfig httpd on
现在，去修改一下httpd.conf文件，开启两个模块的支持，修改之前，我们先来备份一下httpd.conf
[root@apache ~]# cd /etc/httpd/ （因为在编译httpd时，将配置文件的目录即conf下的文件放在了/etc/httpd/下面，所以配置需要去这里。） [root@apache httpd]# cp httpd.conf httpd.conf-10-1.bak [root@apache httpd]# ls extra httpd.conf httpd.conf-10-1.bak magic mime.types original
1 [root@apache httpd]# vim httpd.conf 2 #找到以下两行，取消注释，如果找不到，手动添加进去亦可 3 LoadModule slotmem_shm_module modules/mod_slotmem_shm.so 4 LoadModule slotmem_plain_module modules/mod_slotmem_plain.so
配置完成后，现在检查下配置文件，并启动httpd
[root@apache httpd]# service httpd configtest
Syntax OK
[root@apache httpd]# service httpd start
访问网站，http://192.168.40.184,出现It Works！字样，即表示成功。下面是安装两台tomcat的记录。
Tomcat环节:
1.tomcatA软件的安装。先来安装下jdk。
1 [root@tomcat1 download]# rpm -ivh jdk-7u79-linux-x64.rpm
添加环境变量的文件，新建java.sh文件，内容如下
[root@tomcat1 download]# vim /etc/profile.d/java.sh export JAVA_HOME=/usr/java/jdk1.7.0_79 export PATH=$PATH:$JAVA_HOME/bin
测试一下
1 [root@tomcat1 download]# java -version 2 java version "1.7.0_79" 3 Java(TM) SE Runtime Environment (build 1.7.0_79-b15) 4 Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
2.安装tomcat，解压到/usr/local下
1 [root@tomcat1 download]# tar xf apache-tomcat-7.0.42.tar.gz -C /usr/local/ && cd /usr/local 2 [root@tomcat1 local]# ln -sv apache-tomcat-8.0.26 tomcat 3 [root@tomcat1 local]# ll 4 lrwxrwxrwx 1 root root 21 Oct 1 17:30 tomcat -> apache-tomcat-8.0.26/
现在配置tomcat的环境变量/etc/profile.d/tomcat.sh
1 [root@tomcat1 local]# vim /etc/profile.d/tomcat.sh 2 3 export CATALINA_HOME=/usr/local/tomcat 4 export PATH=$PATH:$CATALINA_HOME/bin
Ctrl + d 键注销一下，再次登陆，试一下命令
[root@tomcat1 local]# catalina.sh version Using CATALINA_BASE: /usr/local/tomcat Using CATALINA_HOME: /usr/local/tomcat Using CATALINA_TMPDIR: /usr/local/tomcat/temp Using JRE_HOME: /usr/java/jdk1.7.0_79 Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar Server version: Apache Tomcat/8.0.26 Server built: Aug 18 2015 11:38:37 UTC Server number: 8.0.26.0 OS Name: Linux OS Version: 2.6.32-431.el6.x86_64 Architecture: amd64 JVM Version: 1.7.0_79-b15 JVM Vendor: Oracle Corporation
方便tomcat服务的管理，我们添加一个脚本，并加入系统服务
1 [root@tomcat1 local]# vim /etc/init.d/tomcat 2 3 #!/bin/sh 4 # Tomcat init script for Linux. 5 # 6 # chkconfig: 2345 96 14 7 # description: The Apache Tomcat servlet/JSP container. 8 CATALINA_HOME=/usr/local/tomcat #注意你的脚本路径 9 export CATALINA_HOME 10 # export CATALINA_OPTS="-Xms128m -Xmx256m" 11 exec $CATALINA_HOME/bin/catalina.sh $* 12 13 [root@tomcat1 local]# chmod +x /etc/init.d/tomcat 14 [root@tomcat1 local]# chkconfig --add tomcat 15 [root@tomcat1 local]# chkconfig tomcat on 16 #之后启动一下，测试脚本是否正常 17 18 [root@tomcat1 local]# service tomcat start 19 Using CATALINA_BASE: /usr/local/tomcat 20 Using CATALINA_HOME: /usr/local/tomcat 21 Using CATALINA_TMPDIR: /usr/local/tomcat/temp 22 Using JRE_HOME: /usr 23 Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar 24 Tomcat started.
检查一下启动的端口号，正常应该会有以下几个监听的端口：8080，8005，然后可以打开浏览器，访问一下http://192.168.40.178.可以看到小猫咪。安装tomcat完成，现在就去修改配置文件，添加一个虚拟主机。
3.虚拟主机配置，文件路径/usr/local/tomcat/conf/server.xml,修改之前先备份此文件。将默认的8080端口修改成80端口，在Engine中，将默认主机localhost修改成www.test.com，并在后面追加一句 jvmRoute = “TomcatA”,最后在文件倒数第三行前（即</Engine>上方）添加一个虚拟主机，如下标红字体。
[root@tomcat1 local]# vim /usr/local/tomcat/conf/server.xml <Connector port="80" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" /> <Engine name="Catalina" defaultHost="www.test.com" jvmRoute="TomcatA"> <Host name="www.test.com" appBase="/web" unpackWARs="true" autoDeploy="true"> <Context path="/" docBase="webapp" reloadable="true"/> </Host> </Engine>
现在检查一下tomcat的配置文件，检查之前需要先停止tomcat，不然会报错。
[root@tomcat1 local]# service tomcat stop [root@tomcat1 local]# service tomcat configtest .... Oct 02, 2015 4:51:35 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 2130 ms
创建虚拟主机文件文档目录并创建测试网页
[root@tomcat1 local]# mkdir -p /web/webapp/ [root@tomcat1 local]# cd /web/webapp/ [root@tomcat1 webapp]# vim index.jsp <%@ page language="java" %> <html> <head><title>TomcatA</title></head> <body> <h1><font color="red">TomcatA </h1> <table align="centre" border="1"> <tr> <td>Session ID</td> <% session.setAttribute("abc","abc"); %> <td><%= session.getId() %></td> </tr> <tr> <td>Created on</td> <td><%= session.getCreationTime() %></td> </tr> </table> </body> </html> [root@tomcat1 webapp]# service tomcat start
启动服务后，可以打开网站，http://192.168.40.178，出现以下说明配置成功了。
另外一台服务器TomcatB也要按照上述的操作部署。一模一样的去部署。仔细，每个环节都不能少了。(注意，唯一不同的是tomcatB服务器的server.xml中的jvmRoute填写的是TomcatB)
四.利用mod_jk实现负载均衡
mod_jk 是一个工作在apache端基于AJP协议与Tomcat通信的连接器，是apache的一个模块，是AJP协议的客户端。（服务端是Tomcat的AJP连接器）
1.下面安装tomcat-connectors
1 [root@apache down]# tar xf tomcat-connectors-1.2.40-src.tar.gz 2 [root@apache down]# cd tomcat-connectors-1.2.40-src/native/ 3 [root@apache native]# ./configure --with-apxs=/usr/local/apache/bin/apxs 4 [root@apache native]# make && make install
2.模块安装完成后，我们可以查看一下。ls /usr/local/apache/modules/ | grep mod_jk .为了便于管理与mod_jk模块相关的配置，在此使用一个专门的配置文件/etc/httpd/extra/httpd-jk.conf来保存相关配置，内容如下（需要新建）：
[root@apache native]# vim /etc/httpd/extra/httpd-jk.conf #Load the mod_jk LoadModule jk_module modules/mod_jk.so #加载mod_jk模块 JkWorkersFile /etc/httpd/extra/workers.properties #配置文件位置 JkLogFile logs/mod_jk.log #日志 JkLogLevel debug #日志级别 JkMount /* lbcluster # 负载均衡器名称 JkMount /jkstatus/ stat1 #（最后单词是个数字1） #状态信息
然后去编辑/etc/httpd/extra/workers.properties文件，内容如下所示：
[root@apache native]# vim /etc/httpd/extra/workers.properties worker.list=lbcluster,stat1 #列表信息 worker.TomcatA.port=8009 #TomcatA实例端口 worker.TomcatA.host=192.168.40.178 #TomcatA主机 worker.TomcatA.type=ajp13 #支持协议 worker.TomcatA.lbfactor=1 #负载均衡权重为1 worker.TomcatB.host=192.168.40.180 worker.TomcatB.port=8009 worker.TomcatB.type=ajp13 worker.TomcatB.lbfactor=1 worker.lbcluster.type = lb #负载均衡work，lb内置类 worker.lbcluster.sticky_session = 0 #会话是否绑定 worker.lbcluster.balance_workers = TomcatA,TomcatB #集群中实例 worker.stat1.type = status #状态信息
现在去修改apache的配置文件，引用上一步我们写入的mod_jk的配置文件，并注释一条信息，如下所示。
[root@apache native]# vim /etc/httpd/httpd.conf #DocumentRoot "/usr/local/apache/htdocs" #在文件的末尾添加下面这行内容 Include /etc/httpd/extra/httpd-jk.conf
下面重启一下httpd服务器，并先停止后端的TomcatA和TomcatB服务器，在启动服务（没有重启功能）。然后打开浏览器，进入测试，访问http://192.168.40.184，这里是apache的网址。刷新网页，则会来回跳转到TomcatA 和TomcatB后端服务器上。如下所示：
通过mod_jk提供的接口，可以查看群集的状态信息。浏览器访问http://192.168.40.184/jkstatus，如下图所示
五.DeltaManager实现session保持
Tomcat 配置
1.修改TomcatA的server.xml配置文件,放在倒数第三行</Engine>上，此处放置的位置定义在Engine容器中，则是对所有主机均启用集群功能，如果定义在某Host主机中，则表示近对此主机启用集群功能。
注意事项：有的文章里配置文件里有这句话：<ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/> ，此选项在tomcat8中被移除了。所以如果你的版本是tomcat8的话，需要注意一下
[root@tomcat1 conf]# vim server.xml <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8"> <Manager className="org.apache.catalina.ha.session.DeltaManager" expireSessionsOnShutdown="false" notifyListenersOnReplication="true"/> <Channel className="org.apache.catalina.tribes.group.GroupChannel"> <Membership className="org.apache.catalina.tribes.membership.McastService" address="228.0.0.4" port="45564" frequency="500" dropTime="3000"/> <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver" address="192.168.40.178" port="4000" autoBind="100" selectorTimeout="5000" maxThreads="6"/> <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter"> <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/> </Sender> <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/> <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/> </Channel> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter="/"/> <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer" tempDir="/tmp/war-temp/" deployDir="/tmp/war-deploy/" watchDir="/tmp/war-listen/" watchEnabled="false"/> <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/> </Cluster>
2.修改TomcatB的server.xml文件，同样在倒数第三行的</Engine>上方添加以下内容。
[root@tomcat2 conf]# vim server.xml <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8"> <Manager className="org.apache.catalina.ha.session.DeltaManager" expireSessionsOnShutdown="false" notifyListenersOnReplication="true"/> <Channel className="org.apache.catalina.tribes.group.GroupChannel"> <Membership className="org.apache.catalina.tribes.membership.McastService" address="228.0.0.4" port="45564" frequency="500" dropTime="3000"/> <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver" address="192.168.40.180" port="4000" autoBind="100" selectorTimeout="5000" maxThreads="6"/> <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter"> <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/> </Sender> <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/> <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/> </Channel> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter="/"/> <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/> <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer" tempDir="/tmp/war-temp/" deployDir="/tmp/war-deploy/" watchDir="/tmp/war-listen/" watchEnabled="false"/> <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/> </Cluster>
分别在tomcatA和tomcatB上 执行以下命令测试,检测配置文件。
[root@tomcat2 conf]# service tomcat configtest ..... Oct 02, 2015 6:21:59 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 2105 ms
此外，所有启用集群功能的web应用程序，其web.xml中都必须添加<distributable/>才能实现集群功能。如果某web应用程序没有自己的web.xml，也可以通过复制默认的web.xml到其WEB-INF目录中实现。
3.修改web.xml
TomcatA:
1 [root@tomcat1 ~]# vim /usr/local/tomcat/conf/web.xml 2 <web-app xmlns="http://xmlns.jcp.org/xml/ns/javaee" 3 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 4 xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee 5 http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd" 6 version="3.1"> 7 8 <distributable/> （只需要添加这一句话即可） 9 10 [root@tomcat1 ~]# mkdir -p /web/webapp/WEB-INF && cp /usr/local/tomcat/conf/web.xml /web/webapp/WEB-INF/
TomcatB：
[root@tomcat2 ~]# vim /usr/local/tomcat/conf/web.xml <web-app xmlns="http://xmlns.jcp.org/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd" version="3.1"> <distributable/> [root@tomcat2 ~]# mkdir -p /web/webapp/WEB-INF && cp /usr/local/tomcat/conf/web.xml /web/webapp/WEB-INF/
4.启动tomcat服务器
TomcatB：
1 [root@tomcat1 ~]# service tomcat start 2 [root@tomcat1 ~]# netstat -aultnp 3 Active Internet connections (servers and established) 4 Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name 5 tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 20397/sshd 6 tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1240/master 7 tcp 0 52 192.168.40.178:22 192.168.30.192:49761 ESTABLISHED 30625/sshd 8 tcp 0 0 :::80 :::* LISTEN 32724/java 9 tcp 0 0 :::22 :::* LISTEN 20397/sshd 10 tcp 0 0 ::1:25 :::* LISTEN 1240/master 11 tcp 0 0 ::ffff:192.168.40.178:4000 :::* LISTEN 32724/java 12 tcp 0 0 ::ffff:127.0.0.1:8005 :::* LISTEN 32724/java 13 tcp 0 0 :::8009 :::* LISTEN 32724/java 14 tcp 0 0 ::ffff:192.168.40.178:43142 ::ffff:192.168.40.180:4000 TIME_WAIT - 15 tcp 0 0 ::ffff:192.168.40.178:43143 ::ffff:192.168.40.180:4000 ESTABLISHED 32724/java 16 tcp 0 0 ::ffff:192.168.40.178:4000 ::ffff:192.168.40.180:48954 ESTABLISHED 32724/java 17 udp 0 0 :::45564 :::* 32724/java
TomcatB：
[root@tomcat2 ~]# service tomcat start [root@tomcat2 ~]# netstat -altpnu Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1159/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1235/master tcp 0 52 192.168.40.180:22 192.168.30.192:49762 ESTABLISHED 24958/sshd tcp 0 0 :::80 :::* LISTEN 26540/java tcp 0 0 :::22 :::* LISTEN 1159/sshd tcp 0 0 ::1:25 :::* LISTEN 1235/master tcp 0 0 ::ffff:192.168.40.180:4000 :::* LISTEN 26540/java tcp 0 0 ::ffff:127.0.0.1:8005 :::* LISTEN 26540/java tcp 0 0 :::8009 :::* LISTEN 26540/java tcp 0 0 ::ffff:192.168.40.180:4000 ::ffff:192.168.40.178:43143 ESTABLISHED 26540/java tcp 0 0 ::ffff:192.168.40.180:48954 ::ffff:192.168.40.178:4000 ESTABLISHED 26540/java udp 0 0 :::45564 :::* 26540/java
5.现在打开浏览器，访问http://192.168.40.184，进行验证。
无论如何刷新，图中的SessionID都是不变的。
通过以上的观察，我们tomcat集群负载功能已经实现，并且配置了DeltaManager得到session的复制。
六.总结
通过以上的安装配置，以及过程中遇到的故障的解决，对apache的以及tomcat得到一些了解和认识。编译安装虽说很麻烦，确实很容易出错。。。。
本文大多数参考freeloda的博客，在原文基础上加入了一点点自己的修改，以及最新版软件使用时的不兼容。原文： “Share your knowledge …” 博客，请务必保留此出处http://freeloda.blog.51cto.com/2033581/1301382
参考资料：关于tomcat8的权威指南：http://wiki.jikexueyuan.com/project/tomcat/
每个人都应是守望者，守望我们的心智，我们的理想，以防它在生活中不知不觉地坠落、被操控和被自己遗忘。。。
