WEB集群 - LNMT集群架构部署zrlog
1. 集群环境说明
LNMP实验场景如下：
-
linux系统版本：CentOS 7.8 X86_64
-
客户端：192.168.20.1
-
nginx负载均衡：主机名：nginx01，地址：192.168.20.20，nginx版本：1.20.1
-
WEB服务器01：主机名：tomcat01，地址：192.168.20.22，tomcat版本：9.0.45
-
WEB服务器02：主机名：tomcat02，地址：192.168.20.23，tomcat版本：9.0.45
-
mysql服务器：主机名：mysql01，地址：192.168.20.50，版本：mariadb-10.5.2
-
redis服务器：主机名redis01，地址：192,168.20.61，版本：3.2.12
-
zrlog：版本：2.2.0
zrlog无法测试session缓存功能，因为其session是存储在数据库中。
注意：所有的软件安装参考前面小节，此处省略。
2. NFS部署
#在tomcat01和tomcat02主机上tomcat程序是以root方式运行的，在NFS服务器上设置把root用户压缩为tomcat用户。
[root@nfs01 ~]# groupadd -g 889 tomcat
[root@nfs01 ~]# useradd -u 889 -g 889 -r tomcat
#创建共享目录
[root@nfs01 ~]# mkdir /data/zrlog
[root@nfs01 ~]# chown tomcat.tomcat /data/zrlog/ -R
[root@nfs01 ~]# vim /etc/exports
/data/zrlog 192.168.20.0/24(rw,all_squash,anonuid=889,anongid=889)
#重启NFS服务
[root@nfs01 ~]# systemctl reload nfs-server.service
3. mysql部署
在数据库上创建数据库和远程连接的用户
[root@mysql01 ~]# mysql -uroot -p123456
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 375
Server version: 10.5.2-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> CREATE DATABASE zrlog;
Query OK, 1 row affected (0.011 sec)
MariaDB [(none)]> GRANT ALL PRIVILEGES ON zrlog.* TO "tomcat"@"192.168.20.%" IDENTIFIED BY "123456";
Query OK, 0 rows affected (0.024 sec)
MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.015 sec)
4. redis部署
#1.安装redis并启动服务
[root@redis01 ~]# yum install redis
#2.修改redis的配置文件：
[root@redis01 ~]# sed -i '/^bind/c bind 127.0.0.1 172.16.1.51' /etc/redis.conf
#3.启动redis服务
[root@redis01 ~]# systemctl start redis
[root@redis01 ~]# ss -ntlp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 192.168.20.61:6379 *:* users:(("redis-server",pid=1939,fd=6))
LISTEN 0 128 127.0.0.1:6379 *:* users:(("redis-server",pid=1939,fd=4))
5. tomcat部署
tomcat01和tomcat02主机的配置完全相同，以tomcat01为例进行说明：
-
在tomcat主配置文件server.xml中增加虚拟主机
#新增加的虚拟主机的域名为zrlog.xuzhichao.com，虚拟主机的根目录为/data/tomcat/zrlog，开启自动部署。 [root@tomcat01 zrlog]# vim /apps/tomcat/conf/server.xml <Host name="zrlog.xuzhichao.com" appBase="/data/tomcat/zrlog" unpackWARs="true" autoDeploy="true"> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="zrlog_access_log" suffix=".txt" pattern="%{X-Forwarded-For}i %l %u %t %r %s %b %{User-Agent}i %{Referer}i" /> <==定义日志格式，显示客户端真实IP地址 </Host> -
下载zrlog软件包，创建/data/tomcat/zrlog目录，将软件包解压到该目录
[root@tomcat01 ~]# mkdir /data/tomcat/zrlog -pv [root@tomcat01 ~]# cd /data/tomcat/zrlog/ #当使用的是war包进行部署时，需要将war包命名为ROOT.war，tomcat会自动部署该应用。 #当使用的是zip包时，需要手动解压到/data/tomcat/zrlog/ROOT目录下。 [root@tomcat01 zrlog]# ll -rw-r--r-- 1 root root 11110760 Jul 6 09:55 zrlog-2.2.0-5e8a51f-release.war [root@tomcat01 zrlog]# mv zrlog-2.2.0-5e8a51f-release.war ROOT.war -
重新启动tomcat服务：
[root@tomcat01 apps]# /apps/tomcat/bin/shutdown.sh && /apps/tomcat/bin/startup.sh Using CATALINA_BASE: /apps/tomcat Using CATALINA_HOME: /apps/tomcat Using CATALINA_TMPDIR: /apps/tomcat/temp Using JRE_HOME: /usr Using CLASSPATH: /apps/tomcat/bin/bootstrap.jar:/apps/tomcat/bin/tomcat-juli.jar Using CATALINA_OPTS: Using CATALINA_BASE: /apps/tomcat Using CATALINA_HOME: /apps/tomcat Using CATALINA_TMPDIR: /apps/tomcat/temp Using JRE_HOME: /usr Using CLASSPATH: /apps/tomcat/bin/bootstrap.jar:/apps/tomcat/bin/tomcat-juli.jar Using CATALINA_OPTS: Tomcat started. -
重新启动tomcat后，会对指定的目录中的war包进行自动解压部署
[root@tomcat01 zrlog]# ll total 10852 drwxr-x--- 9 root root 126 Jul 6 10:04 ROOT -rw-r--r-- 1 root root 11110760 Jul 6 09:55 ROOT.war #查看日志： [root@tomcat01 zrlog]# tail -f /apps/tomcat/logs/catalina.out 06-Jul-2021 15:39:47.643 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/data/tomcat/zrlog/ROOT.war] has finished in [1,287] ms 06-Jul-2021 15:39:47.660 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"] 06-Jul-2021 15:39:47.790 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [3235] milliseconds -
zrlog的图片目录为/data/tomcat/zrlog/ROOT/attached/，把NFS的共享目录挂载到此目录：
[root@tomcat01 apps]# mount -t nfs 192.168.20.30:/data/zrlog /data/tomcat/zrlog/ROOT/attached/ [root@tomcat01 apps]# df Filesystem 1K-blocks Used Available Use% Mounted on 192.168.20.30:/data/zrlog 154057344 33408 154023936 1% /data/tomcat/zrlog/ROOT/attached -
客户端分别访问tomcat进行连接数据库：
6. nginx负载均衡部署
nginx负载均衡的配置文件为：
[root@nginx01 ~]# cat /etc/nginx/conf.d/zrlog.xuzhichao.conf
upstream tomcat {
server 192.168.20.22:8080 weight=1 fail_timeout=5s max_fails=3;
server 192.168.20.23:8080 weight=1 fail_timeout=5s max_fails=3;
}
server {
listen 443 ssl;
listen 80;
server_name zrlog.xuzhichao.com;
access_log /var/log/nginx/access_zrlog.log access_json;
ssl_certificate /etc/nginx/certs/xuzhichao.crt;
ssl_certificate_key /etc/nginx/certs/xuzhichao.key;
ssl_session_cache shared:ssl_cache:30m;
ssl_session_timeout 10m;
valid_referers none blocked server_names *.b.com b.* ~\.baidu\. ~\.google\.;
if ( $invalid_referer ) {
return 403;
}
location / {
if ( $scheme = http ) {
rewrite /(.*) https://zrlog.xuzhichao.com/$1 permanent;
}
proxy_pass http://tomcat;
include proxy_params;
}
}
[root@nginx01 ~]# cat /etc/nginx/proxy_params
proxy_set_header host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 30;
proxy_send_timeout 60;
proxy_read_timeout 60;
proxy_buffering on;
proxy_buffer_size 64k;
proxy_buffers 4 64k;
#重启nginx
[root@nginx01 ~]# systemctl reload nginx.service
7. 客户端访问
客户端访问：https://zrlog.xuzhichao.com/可以正常打开zrlog页面。
验证负载均衡成功的方法，新打开一个页面，在两个tomcat主机上分别查看访问日志，两个同时产生访问日志：
[root@tomcat01 apps]# tail -f /apps/tomcat/logs/zrlog_access_log.2021-07-06.txt
192.168.20.1 - - [06/Jul/2021:23:19:13 +0800] GET /admin/login HTTP/1.0 302 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/post/2
192.168.20.1 - - [06/Jul/2021:23:19:13 +0800] GET /admin/static/css/main.6aa21713.chunk.css HTTP/1.0 200 1220714 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/admin/index
192.168.20.1 - - [06/Jul/2021:23:19:13 +0800] GET /admin/static/js/9.4b2aeb10.chunk.js HTTP/1.0 200 382645 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/admin/index
192.168.20.1 - - [06/Jul/2021:23:19:13 +0800] GET /admin/static/css/6.f8767b4f.chunk.css HTTP/1.0 200 551 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/admin/index
192.168.20.1 - - [06/Jul/2021:23:19:13 +0800] GET /admin/static/js/0.7b496ee5.chunk.js HTTP/1.0 200 63660 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/admin/index
192.168.20.1 - - [06/Jul/2021:23:19:13 +0800] GET /admin/static/js/2.70e8786e.chunk.js HTTP/1.0 200 94827 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/admin/index
192.168.20.1 - - [06/Jul/2021:23:19:13 +0800] GET /admin/static/js/25.bbae9f3c.chunk.js HTTP/1.0 200 3960 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/admin/index
192.168.20.1 - - [06/Jul/2021:23:19:13 +0800] GET /admin/vendors/js/fonts.js HTTP/1.0 200 2515 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/admin/index
192.168.20.1 - - [06/Jul/2021:23:19:14 +0800] GET /api/admin/serverInfo HTTP/1.0 200 5720 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/admin/index
[root@tomcat02 apps]# tail -f /apps/tomcat/logs/zrlog_access_log.2021-07-06.txt
192.168.20.1 - - [06/Jul/2021:23:19:13 +0800] GET /admin/index HTTP/1.0 200 10516 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 -
192.168.20.1 - - [06/Jul/2021:23:19:13 +0800] GET /admin/static/js/main.d0c32646.chunk.js HTTP/1.0 200 8549 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/admin/index
192.168.20.1 - - [06/Jul/2021:23:19:13 +0800] GET /admin/static/js/6.4ff1aa8f.chunk.js HTTP/1.0 200 12132 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/admin/index
192.168.20.1 - - [06/Jul/2021:23:19:13 +0800] GET /admin/static/js/5.4f016702.chunk.js HTTP/1.0 200 280718 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/admin/index
192.168.20.1 - - [06/Jul/2021:23:19:13 +0800] GET /admin/static/js/1.15016a67.chunk.js HTTP/1.0 200 68316 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/admin/index
192.168.20.1 - - [06/Jul/2021:23:19:13 +0800] GET /admin/static/js/13.5922a7b6.chunk.js HTTP/1.0 200 21440 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/admin/index
192.168.20.1 - - [06/Jul/2021:23:19:14 +0800] GET /api/admin/user/basicInfo HTTP/1.0 200 173 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/admin/index
192.168.20.1 - - [06/Jul/2021:23:19:14 +0800] GET /api/admin/statisticsInfo HTTP/1.0 200 101 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/admin/index
192.168.20.1 - - [06/Jul/2021:23:19:14 +0800] GET /assets/images/default-portrait.gif HTTP/1.0 200 2926 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 https://zrlog.xuzhichao.com/admin/index
NFS服务器上存在上传的图片资源：
[root@nfs01 ~]# tree /data/zrlog/image/
/data/zrlog/image/
└── 20210706
├── 20210706160155_616.png
├── 20210706160155_616_thumbnail.png
├── 20210706230030_967.png
├── 20210706230923_993.png
└── 20210706230923_993_thumbnail.png
8. tomcat+redis实现会话共享
使用上面的实验环境验证tomcat+redis实现session的共享。
两台tomcat主机的配置相同，测试页不同，可以显示测试效果。
-
在tomcat主机上新增加一个虚拟主机session.xuzhichao.com.
<Host name="session.xuzhichao.com" appBase="/data/tomcat/session" unpackWARs="true" autoDeploy="true"> </Host> -
新建业务目录，添加测试代码
[root@tomcat02 apps]# mkdir /data/tomcat/session/ROOT -p [root@tomcat02 apps]# cat /data/tomcat/session/ROOT/index.jsp <body> <% //HttpSession session = request.getSession(true); System.out.println(session.getCreationTime()); out.println("<br> web02 SESSION ID:" + session.getId() + "<br>"); out.println("Session created time is :" + session.getCreationTime() + "<br>"); %> </body> [root@tomcat01 apps]# cat /data/tomcat/session/ROOT/index.jsp <body> <% //HttpSession session = request.getSession(true); System.out.println(session.getCreationTime()); out.println("<br> web01 SESSION ID:" + session.getId() + "<br>"); out.println("Session created time is :" + session.getCreationTime() + "<br>"); %> </body> -
nginx负载均衡新增配置文件：
[root@nginx01 ~]# cat /etc/nginx/conf.d/session.xuzhichao.conf upstream tomcat_session { server 192.168.20.22:8080 weight=1 fail_timeout=5s max_fails=3; server 192.168.20.23:8080 weight=1 fail_timeout=5s max_fails=3; } server { listen 443 ssl; listen 80; server_name session.xuzhichao.com; access_log /var/log/nginx/access_session.log access_json; ssl_certificate /etc/nginx/certs/xuzhichao.crt; ssl_certificate_key /etc/nginx/certs/xuzhichao.key; ssl_session_cache shared:ssl_cache:30m; ssl_session_timeout 10m; valid_referers none blocked server_names *.b.com b.* ~\.baidu\. ~\.google\.; if ( $invalid_referer ) { return 403; } location / { if ( $scheme = http ) { rewrite /(.*) https://session.xuzhichao.com/$1 permanent; } proxy_pass http://tomcat_session; include proxy_params; } } [root@nginx01 ~]# systemctl reload nginx.service -
下载安装TomcatClusterRedisSessionManager组件：
[root@tomcat01 ~]# wget https://github.com/ran-jit/tomcat-cluster-redis-session-manager/releases/download/3.0.3/tomcat-cluster-redis-session-manager.zip [root@tomcat01 ~]# unzip tomcat-cluster-redis-session-manager.zip -
将下载的包移动到tomcat/lib目录：
[root@tomcat01 ~]# cp tomcat-cluster-redis-session-manager/lib/* /apps/tomcat/lib/ -
将redis-data-cache.properties文件移动到tomcat/conf目录，并增加redis参数：
[root@tomcat01 ~]# cp tomcat-cluster-redis-session-manager/conf/redis-data-cache.properties /apps/tomcat/conf/ [root@tomcat01 ~]# vim /apps/tomcat/conf/redis-data-cache.properties ... redis.hosts=192.168.20.61:6379 <==redis服务器的地址 #redis.password=123456 <==redis的密码 redis.cluster.enabled=false -
在tomcat/conf/context.xml中增加以下两行：
[root@tomcat01 ~]# vim /apps/tomcat/conf/context.xml <Context> <Valve className="tomcat.request.session.redis.SessionHandlerValve"/> <Manager className="tomcat.request.session.redis.SessionManager" /> </Context> -
重启tomcat服务：
[root@tomcat01 apps]# /apps/tomcat/bin/shutdown.sh && /apps/tomcat/bin/startup.sh -
客户端测试访问https://session.xuzhichao.com/
发现两台tomcat主机的session信息已经一致：
而且与redis服务器上记录的session信息也一致：
[root@redis01 ~]# redis-cli
127.0.0.1:6379> KEYS *
1) "2749E3AC18F8D75E5772F8E035F45572"
127.0.0.1:6379>
