企业应用完美架构
此系统架构仅映射内网VIP的80及443端口于外网的Juniper防火墙下,其它端口均关闭,内网所有机器均关闭iptables及ipfw防火墙;外网DNS指向即通过Juniper映射出来的外网地址,而此映射的地址对映的其实是内网VIP地址。这里说下端口的问题,有的朋友可能会很疑惑,这样映射端口行不?通过项目实践得知,这样完全是可行的,php-cgi需要的9000端口及MySQL的3306端口均可走内网,完全不影响业务系统的运行。
另外,我维护的电子商务网站并发大约在1000左右,此时,Nginx+Apache集群运行得非常稳定,尤其是apache,并没有想象中那般弱;其实,在内存足够(>=8G)的情况,测试时不连数据库的话,单台apache+php5能顶得住6000并发,而且相当稳定。在网站升级架构方面,我不赞成全面淘汰生级,锦上添花式的升级会更好。
第一部分:Nginx+Keepalived的说明及环境说明
喜欢看我博客或文章的朋友都知道,我一直主力推崇Nginx+Keepalived作web的负载均衡高可用架构,并积极将其用于项目方案中;Nginx负载均衡作服务器遇到的故障一般有①服务器网线松动等网络故障;②服务器硬件故障从而crash;③nginx服务死掉;遇到前二者情况,keeaplived是能起到HA的作用的;然而遇到③种情况就没有办法了,但可以通过shell监控解决这问题,从而实现真正意义上的负载均衡高可用。此篇的最新更新时间为2010年6月25号,下面将其安装步骤详细说明下:
环境:
1. centos5.3(64位)、nginx-0.7.51、keepalived-1.1.15
2. 主nginx负载均衡器:192.168.0.154
3. 辅nginx负载均衡器:192.168.9.155
4. vip:192.168.0.188
先安装Nginx负载均衡器,nginx负载的配置就用一般的模板来配置了
1. #添加运行nginx的用户和组www
2. groupadd www
3. useradd -g www www
4. wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-7.8.tar.gz
5. tar zxvf pcre-7.8.tar.gz
6. cd pcre-7.8/
7. ./configure
8. make && make install
9. wget http://sysoev.ru/nginx/nginx-0.7.51.tar.gz
10. tar zxvf nginx-0.7.51.tar.gz
11. cd nginx-0.7.51/
12. ./configure --user=www --group=www --prefix=/usr/local/webserver/nginx --with-http_stub_status_module --with-http_ssl_module
13. make && make install
配置nginx负载均衡器的配置文件vim /usr/local/nginx/conf/nginx.conf,此篇文章仅仅只是我的某项目的配置文档,纯80转发;如果对nginx配置有https要求的可参考张宴的相关文章。
14. user www www;
15. worker_processes 8;
16.
17. pid /usr/local/nginx/logs/nginx.pid;
18. worker_rlimit_nofile 65535;
19.
20. events
21. {
22. use epoll;
23. worker_connections 65535;
24. }
25. http{
26. include mime.types;
27. default_type application/octet-stream;
28. server_names_hash_bucket_size 128;
29. client_header_buffer_size 32k;
30. large_client_header_buffers 4 32k;
31. client_max_body_size 8m;
32. sendfile on;
33. tcp_nopush on;
34. keepalive_timeout 60;
35. tcp_nodelay on;
36. fastcgi_connect_timeout 300;
37. fastcgi_send_timeout 300;
38. fastcgi_read_timeout 300;
39. fastcgi_buffer_size 64k;
40. fastcgi_buffers 4 64k;
41. fastcgi_busy_buffers_size 128k;
42. fastcgi_temp_file_write_size 128k;
43. gzip on;
44. gzip_min_length 1k;
45. gzip_buffers 4 16k;
46. gzip_http_version 1.0;
47. gzip_comp_level 2;
48. gzip_types text/plain application/x-javascript text/css application/xml;
49. gzip_vary on;
50.
51. upstream backend
52. {
53. server 192.168.1.102:80;
54. server 192.168.1.103:80;
55. server 192.168.1.105:80;
56. }
57. server {
58. listen 80;
59. server_name www.yuhongchun027.com;
60. location / {
61. root /var/www ;
62. index index.jsp index.htm index.html;
63. proxy_redirect off;
64. proxy_set_header Host $host;
65. proxy_set_header X-Real-IP $remote_addr;
66. proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
67. proxy_pass http://backend;
68. }
69.
70. location /nginx {
71. access_log on;
72. auth_basic "NginxStatus";
73. auth_basic_user_file /usr/local/nginx/htpasswd;
74. }
75.
76. log_format access '$remote_addr - $remote_user [$time_local] "$request" '
77. '$status $body_bytes_sent "$http_referer" '
78. '"$http_user_agent" $http_x_forwarded_for';
79. access_log /var/log/access.log access;
80.
81. }
82. }
小节:
第一部分和第二部分讲的是如何通过安装Nginx来达到负载均衡后端web集群的过程,Nginx能实现自动切换后端有故障的web服务器;但Nginx负载均衡器出了问题怎么办呢,它们之间是如何实现无故障转移的呢?
安装keepalived,并将其做成服务模式,方便以后调试。
1. wget http://www.keepalived.org/software/keepalived-1.1.15.tar.gz
2. #tar zxvf keepalived-1.1.15.tar.gz
3. #cd keepalived-1.1.15
4. #./configure --prefix=/usr/local/keepalived
5. #make
6. #make install
7. #cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
8. #cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
9. #cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
10. #mkdir /etc/keepalived
11. #cd /etc/keepalived/
12.
13. vim keepalived.conf
14. ! Configuration File for keepalived
15. global_defs {
16. notification_email {
17. yuhongchun027@163.com
18. }
19. notification_email_from keepalived@chtopnet.com
20. smtp_server 127.0.0.1
21. smtp_connect_timeout 30
22. router_id LVS_DEVEL
23. }
24. vrrp_instance VI_1 {
25. state MASTER
26. interface eth0
27. virtual_router_id 51
28. mcast_src_ip 192.168.0.154 <==主nginx的IP地址
29. priority 100
30. advert_int 1
31. authentication {
32. auth_type PASS
33. auth_pass chtopnet
34. }
35. virtual_ipaddress {
36. 192.168.0.188 <==vip地址
37. }
38. }
39. #service keepalived start
我们来看一下日志:
40. [root@ltos ~]# tail /var/log/messages
41. Oct 6 03:25:03 ltos avahi-daemon[2306]: Registering new address record for 192.168.0.188 on eth0.
42. Oct 6 03:25:03 ltos avahi-daemon[2306]: Registering new address record for 192.168.0.154 on eth0.
43. Oct 6 03:25:03 ltos avahi-daemon[2306]: Registering HINFO record with values 'I686'/'LINUX'.
44. Oct 6 03:25:23 ltos avahi-daemon[2306]: Withdrawing address record for fe80::20c:29ff:feb9:eeab on eth0.
45. Oct 6 03:25:23 ltos avahi-daemon[2306]: Withdrawing address record for 192.168.0.154 on eth0.
46. Oct 6 03:25:23 ltos avahi-daemon[2306]: Host name conflict, retrying with <ltos-31>
很显然vrrp已经启动,我们还可以通过命令来检查
47. [root@ltos html]# ip addr
48. 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
49. link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
50. inet 127.0.0.1/8 scope host lo
51. inet6 ::1/128 scope host
52. valid_lft forever preferred_lft forever
53. 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
54. link/ether 00:0c:29:ba:9b:e7 brd ff:ff:ff:ff:ff:ff
55. inet 192.168.0.154/24 brd 192.168.0.255 scope global eth0
56. inet 192.168.0.188/32 scope global eth0
57. inet6 fe80::20c:29ff:feba:9be7/64 scope link
58. valid_lft forever preferred_lft forever
59. 3: sit0: <NOARP> mtu 1480 qdisc noop
60. link/sit 0.0.0.0 brd 0.0.0.0
说明vip已经启动,这样主服务器就配置好了,辅机的配置大致一样,除了配置文件有少部分的变化,下面贴出辅机的配置文件:
61. ! Configuration File for keepalived
62. global_defs {
63. notification_email {
64. yuhongchun027@163.com
65. }
66. notification_email_from keepalived@chtopnet.com
67. smtp_server 127.0.0.1
68. smtp_connect_timeout 30
69. router_id LVS_DEVEL
70. }
71. vrrp_instance VI_1 {
72. state BACKUP
73. interface eth0
74. virtual_router_id 51
75. mcast_src_ip 192.168.0.155 <==辅nginx的IP的地址
76. priority 100
77. advert_int 1
78. authentication {
79. auth_type PASS
80. auth_pass chtopnet
81. }
82. virtual_ipaddress {
83. 192.168.0.188
84. }
85. }
针对Nginx+Keepalived,编写nginx监控脚本nginx_pid.sh,此脚本思路其实也很简单,即放置在后台一直监控nginx进程;如进程消失,尝试重启nginx,如是失败则立即停掉本机的keepalived服务,让另一台负载均衡器接手,此脚本直接从生产环境下载:
1. vim /root/nginx_pid.sh
2. #!/bin/bash
3. while :
4. do
5. nginxpid=`ps -C nginx --no-header | wc -l`
6. if [ $nginxpid -eq 0 ];then
7. /usr/local/nginx/sbin/nginx
8. sleep 5
9. if [ $nginxpid -eq 0 ];then
10. /etc/init.d/keepalived stop
11. fi
12. fi
13. sleep 5
14. done
然后置于后台运行 sh /root/nginx_pid.sh &,这种写法是错误的,这样你用root用户logout后,此进程会消失;正确写法为nohup/bin/bash /root/nginx_pid.sh &,附带下注释:如果你正在运行一个进程,而且你觉得在退出帐户时该进程还不会结束,那么可以使用nohup命令。该命令可以在你退出root帐户之后继续运行相应的进程。nohup就是不挂起的意思( no hang up),哈哈,差点老马失蹄了。
后记:
我的线上环境网络非常复杂,这也是LVS+Keepalived失败的原因。目前此套架构在1000并发的电子商务网站非常稳定,带来的直接影响就是nginx_backup一直处于闲置状态。相对于张宴的双机轮询而言,我感觉他的可能更加完美,因为目前我的Nginx仅仅只做了负载均衡器,如果以后有机会我会尝试做负载均衡器/反向代理加速。