nginx配置负载

一.系统优化

1、修改/etc/sysctl.conf，优化tcp连接数

net.ipv4.tcp_fin_timeout = 30

net.ipv4.tcp_keepalive_time = 1200

net.ipv4.tcp_syncookies = 1

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_tw_recycle = 1

net.ipv4.ip_local_port_range = 1024 65000

net.ipv4.tcp_max_syn_backlog = 8192

net.ipv4.tcp_max_tw_buckets = 5000

2、修改/etc/security/limits.conf，修改最大文件打开数为65535

添加如下两行：

*      soft  nofile       65535

*      hard nofile       65535

*      soft  nproc       16384

*      hard nproc       16384

二.安装JDK

chmod +x jdk-6u38-linux-x64.bin

# ./jdk-6u38-linux-x64.bin

 

#添加环境变量

# vi /etc/profile

export JAVA_HOME=/kjt/java/jdk1.6.0_38

export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar

export PATH=$JAVA_HOME/bin:$PATH

 

#使环境变量生效

# source /etc/profile

 

#查看java版本

# java -version

java version "1.6.0_38"

Java(TM) SE Runtime Environment (build 1.6.0_27-b07)

Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)

 

三.安装Tomcat

1、确保防火墙已关闭

#service iptables stop

#chkconfig iptables off

#vi /etc/selinux/config

selinux=disabled

2、安装tomcat

# tar zxvf apache-tomcat-6.0.36.tar.gz

# cd apache-tomcat-6.0.36

#修改/bin/catalina.sh，在cygwin=false前加路径

export JAVA_HOME=/kjt/java/jdk1.6.0_38/

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export TOMCAT_HOME=/kjt/java/apache-tomcat-6.0.36

export CATALINA_HOME=/kjt/java/apache-tomcat-6.0.36

export CATALINA_BASE=/kjt/java/apache-tomcat-6.0.36

 

3、修改/conf/server.xml，做tomcat线程池优化

    <Connector port="8081" protocol="HTTP/1.1"

              connectionTimeout="15000" maxThreads="500" minSpareThreads="50" maxSpareTheads="250" maxProcessors="600"

              acceptCount="1000" enableLookups="false" compression="on"

              redirectPort="8443" URIEncoding="utf-8" useBodyEncodingForURI="true"/>   

<!-- A "Connector" using the shared thread pool-->

4、启动tomcat

cd /kjt/java/apache-tomcat-6.0.35/bin

./startup

 

5、查看tomcat进程

ps –ef|grep tomcat

四.Nginx配置

1.确定Nginx所需pcre库、openssl库等依赖包已经安装，若没有安装，安装nginx过程中将报错。

2.安装Nginx

tar zxvf nginx-1.0.15.tar.gz

cd nginx-1.0.15/

./configure  --prefix=/kjt/app/nginx --with-http_stub_status_module --with-http_ssl_module

make

make install

 

3.Nginx配置文件如下：

4.测试配置文件

/kjt/app/nginx/sbin/nginx -t

显示以下信息为正确的

the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok

configuration file /usr/local/nginx/conf/nginx.conf test is successful

 

5、启动nginx服务

/kjt/app/nginx/sbin/nginx

 

6、查看nginx进程

ps -ef|grep nginx

 

五.安装配置Keepalived

1、安装keepalived

tar -zxvf keepalived-1.2.7.tar.gz

cd keepalived-1.2.7

./configure --prefix=/kjt/app/keepalived

make

make install

 

2、将keepalived设置为开机启动

cp /kjt/app/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/

cp/kjt/app/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

cp /kjt/app/keepalived/sbin/keepalived /usr/sbin/

chkconfig --add keepalived

chkconfig keepalived on

 

3.修改配置文件为以下内容:

mkdir /etc/keepalived

cp /kjt/app/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

vim  /etc/keepalived/keepalived.conf

 

 

! Configuration File for keepalived

 

global_defs {

        router_id Nginx_Id_1

}

 

vrrp_script Monitor_Nginx {

    script "/kjt/app/keepalived/etc/keepalived/samples/monitor_nginx.sh "  #脚本存放位置

interval 2     #检查间隔为2秒

    weight 2

}

 

vrrp_instance VI_1 {

        state BACKUP  #设置nopreempt，表示之前挂掉的master重新恢复后不再切换为master，直接成为BACKUP，nopreempt只能配置在BACKUP状态下

        nopreempt

    interface eth0

    virtual_router_id 51

        priority 100    #优先级较高的成为MASTER

        advert_int 5   #健康检查间隔为5秒

 

    authentication {

        auth_type PASS

        auth_pass kjt2013

    }

 

    virtual_ipaddress {

       10.46.86.60

    }

    track_script {

        Monitor_Nginx

    }

}

注：monitor_nginx.sh为监控nginx进程的脚本,内容如下：当检测到nginx主进程挂掉之后，系统将自动重启nginx，若nginx主进程重启失败，延迟5秒后杀掉keepalived进程，进行切换。

 

 

#!/bin/bash

if [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ]

then

        /kjt/app/nginx/sbin/nginx

        sleep 5

 

        if [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ]

        then

                killall keepalived

        fi

fi

 

5.启动keepalived

service keepalived start

 

注:备机的keepalived的安装和上面一样，只要把配置文件中的priority改为小于100的数即可

 

6、配置Keepalived监控脚本，每分钟执行一次，当检测到原先MASTER上的Keepalived进程挂掉之后，将重启Keepalived进程

vi  /root/shell/keepalived.sh

touch  /root/shell/keepalived.log

chmod 755 *

新增计划任务

crontab –e

* * * * * (sh /root/shell/keepalived.sh >> /root/shell/keepalived.log)

 

六．测试步骤

1．访问VIP看是否能够正常访问后端的tomcat

2．停止其中一个tomcat看是否能将访问转到另一台上

3．停止两台nginx上任何一个nginx进程看监控进程脚本是否会自动启动nginx

VRRP_Script(Monitor_Nginx) timed out

VRRP_Script(Monitor_Nginx) succeeded

4．停止任何一台nginx上的keepalived进程看另一台是否接管vip

比如停止Master上的keepalived,例如如下killall keepalived,查看BACKUP机器是否已经接管，如果BACKUP接管后，BACKUP机器日志会是出下情况

tail  -f  /var/log/messages

Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE

Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.

Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.46.86.60

 

MASTER机器上日志会显示

Keepalived_vrrp: Terminating VRRP child process on signal

Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.

 

现在把原先MASTER上的Keepalived重新启动，会看到原先MASTER直接成为BACKUP，不再做一次切换。

 

现在的BACKUP日志如下：

Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert

Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE

Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.

 

Master日志如下：

Keepalived_vrrp: VRRP_Script(Monitor_Nginx) succeeded

Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE

Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.

Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.46.86.60

 

 

一.系统优化

1、修改/etc/sysctl.conf，优化tcp连接数

net.ipv4.tcp_fin_timeout = 30

net.ipv4.tcp_keepalive_time = 1200

net.ipv4.tcp_syncookies = 1

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_tw_recycle = 1

net.ipv4.ip_local_port_range = 1024 65000

net.ipv4.tcp_max_syn_backlog = 8192

net.ipv4.tcp_max_tw_buckets = 5000

2、修改/etc/security/limits.conf，修改最大文件打开数为65535

添加如下两行：

*      soft  nofile       65535

*      hard nofile       65535

*      soft  nproc       16384

*      hard nproc       16384

二.安装JDK

chmod +x jdk-6u38-linux-x64.bin

# ./jdk-6u38-linux-x64.bin

 

#添加环境变量

# vi /etc/profile

export JAVA_HOME=/kjt/java/jdk1.6.0_38

export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar

export PATH=$JAVA_HOME/bin:$PATH

 

#使环境变量生效

# source /etc/profile

 

#查看java版本

# java -version

java version "1.6.0_38"

Java(TM) SE Runtime Environment (build 1.6.0_27-b07)

Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode)

 

三.安装Tomcat

1、确保防火墙已关闭

#service iptables stop

#chkconfig iptables off

#vi /etc/selinux/config

selinux=disabled

2、安装tomcat

# tar zxvf apache-tomcat-6.0.36.tar.gz

# cd apache-tomcat-6.0.36

#修改/bin/catalina.sh，在cygwin=false前加路径

export JAVA_HOME=/kjt/java/jdk1.6.0_38/

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export TOMCAT_HOME=/kjt/java/apache-tomcat-6.0.36

export CATALINA_HOME=/kjt/java/apache-tomcat-6.0.36

export CATALINA_BASE=/kjt/java/apache-tomcat-6.0.36

 

3、修改/conf/server.xml，做tomcat线程池优化

    <Connector port="8081" protocol="HTTP/1.1"

              connectionTimeout="15000" maxThreads="500" minSpareThreads="50" maxSpareTheads="250" maxProcessors="600"

              acceptCount="1000" enableLookups="false" compression="on"

              redirectPort="8443" URIEncoding="utf-8" useBodyEncodingForURI="true"/>   

<!-- A "Connector" using the shared thread pool-->

4、启动tomcat

cd /kjt/java/apache-tomcat-6.0.35/bin

./startup

 

5、查看tomcat进程

ps –ef|grep tomcat

四.Nginx配置

1.确定Nginx所需pcre库、openssl库等依赖包已经安装，若没有安装，安装nginx过程中将报错。

2.安装Nginx

tar zxvf nginx-1.0.15.tar.gz

cd nginx-1.0.15/

./configure  --prefix=/kjt/app/nginx --with-http_stub_status_module --with-http_ssl_module

make

make install

 

3.Nginx配置文件如下：

4.测试配置文件

/kjt/app/nginx/sbin/nginx -t

显示以下信息为正确的

the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok

configuration file /usr/local/nginx/conf/nginx.conf test is successful

 

5、启动nginx服务

/kjt/app/nginx/sbin/nginx

 

6、查看nginx进程

ps -ef|grep nginx

 

五.安装配置Keepalived

1、安装keepalived

tar -zxvf keepalived-1.2.7.tar.gz

cd keepalived-1.2.7

./configure --prefix=/kjt/app/keepalived

make

make install

 

2、将keepalived设置为开机启动

cp /kjt/app/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/

cp/kjt/app/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

cp /kjt/app/keepalived/sbin/keepalived /usr/sbin/

chkconfig --add keepalived

chkconfig keepalived on

 

3.修改配置文件为以下内容:

mkdir /etc/keepalived

cp /kjt/app/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

vim  /etc/keepalived/keepalived.conf

 

! Configuration File for keepalived

 

global_defs {

        router_id Nginx_Id_1

}

 

vrrp_script Monitor_Nginx {

    script "/kjt/app/keepalived/etc/keepalived/samples/monitor_nginx.sh "  #脚本存放位置

interval 2     #检查间隔为2秒

    weight 2

}

 

vrrp_instance VI_1 {

        state BACKUP  #设置nopreempt，表示之前挂掉的master重新恢复后不再切换为master，直接成为BACKUP，nopreempt只能配置在BACKUP状态下

        nopreempt

    interface eth0

    virtual_router_id 51

        priority 100    #优先级较高的成为MASTER

        advert_int 5   #健康检查间隔为5秒

 

    authentication {

        auth_type PASS

        auth_pass kjt2013

    }

 

    virtual_ipaddress {

       10.46.86.60

    }

    track_script {

        Monitor_Nginx

    }

}

注：monitor_nginx.sh为监控nginx进程的脚本,内容如下：当检测到nginx主进程挂掉之后，系统将自动重启nginx，若nginx主进程重启失败，延迟5秒后杀掉keepalived进程，进行切换。

 

#!/bin/bash

if [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ]

then

        /kjt/app/nginx/sbin/nginx

        sleep 5

 

        if [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ]

        then

                killall keepalived

        fi

fi

 

5.启动keepalived

service keepalived start

 

注:备机的keepalived的安装和上面一样，只要把配置文件中的priority改为小于100的数即可

 

6、配置Keepalived监控脚本，每分钟执行一次，当检测到原先MASTER上的Keepalived进程挂掉之后，将重启Keepalived进程

vi  /root/shell/keepalived.sh

touch  /root/shell/keepalived.log

chmod 755 *

新增计划任务

crontab –e

* * * * * (sh /root/shell/keepalived.sh >> /root/shell/keepalived.log)

 

六．测试步骤

1．访问VIP看是否能够正常访问后端的tomcat

2．停止其中一个tomcat看是否能将访问转到另一台上

3．停止两台nginx上任何一个nginx进程看监控进程脚本是否会自动启动nginx

VRRP_Script(Monitor_Nginx) timed out

VRRP_Script(Monitor_Nginx) succeeded

4．停止任何一台nginx上的keepalived进程看另一台是否接管vip

比如停止Master上的keepalived,例如如下killall keepalived,查看BACKUP机器是否已经接管，如果BACKUP接管后，BACKUP机器日志会是出下情况

tail  -f  /var/log/messages

Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE

Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.

Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.46.86.60

 

MASTER机器上日志会显示

Keepalived_vrrp: Terminating VRRP child process on signal

Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.

 

现在把原先MASTER上的Keepalived重新启动，会看到原先MASTER直接成为BACKUP，不再做一次切换。

 

现在的BACKUP日志如下：

Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert

Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE

Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.

 

Master日志如下：

Keepalived_vrrp: VRRP_Script(Monitor_Nginx) succeeded

Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE

Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.

Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.46.86.60