基于keepalived、redis sentinel的多实例redis集群【1】---路由以及哨兵配置
硬件
| 机器名 | IP | 作用 |
| master | 192.168.0.2 | redis的master服务器,两个主实例 |
| slave1 | 192.168.0.3 | redis的slave服务器，两个从实例 |
| slave2 | 192.168.0.4 | redis的slave服务器，两个从实例 |
| route1 | 192.168.0.5【虚拟IP：192.168.0.7】 | keepalived和redis sentinel服务器，承载写redis的VIP【虚拟ip】，做写的双机热备的主master指定，redis哨兵的安装节点1 |
| route2 | 192.168.0.6【虚拟IP：192.168.0.8】 | keepalived和redis sentinel服务器，承载读redis的VIP，做读的负载均衡和写的双机热备的master备份路由指定，redis哨兵的安装节点2 |
route1
1.安装redis在route1上，安装路径/usr/local/redis/
2.在redis安装路径下创建scripts目录，将需要的脚本复制到此处：
| 1 | RunCmd.py | 基础功能模块，提供redis服务超时检查 |
| 2 | master_config_set.py | 将master的save参数配置为空 |
| 3 | redischeck.py | 检查master的redis服务状态 |
| 4 | slave_config_set.py | 将slave的save参数配置为特定值 |
| 5 | weightchange.py | 调整读的redis服务在keepalived的权重 |
详细的keepalived配置，
! Configuration File for keepalived
global_defs {
notification_email {
接收邮箱
}
notification_email_from 发送邮箱
smtp_server 邮件服务器
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER
interface eth1
lvs_sync_daemon_inteface eth1
virtual_router_id 100
priority 160
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.0.7
}
}
vrrp_instance VI_2 {
state BACKUP
interface eth1
lvs_sync_daemon_inteface eth1
virtual_router_id 101
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.0.
}
}
virtual_server 192.168.0.7 6379 {
delay_loop 3
lb_algo rr
lb_kind DR
#nat_mask 255.255.255.0
persistence_timeout 15
protocol TCP
real_server 192.168.0.2 6379 {
weight 8
notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.2 6379"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.2 6379"
misc_timeout 5
misc_dynamic
}
}
real_server 192.168.0.3 6379 {
weight 3
notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.3 6379"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.3 6379"
misc_timeout 5
misc_dynamic
}
}
real_server 192.168.0.4 6379 {
weight 3
notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.4 6379"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.4 6379"
misc_timeout 5
msic_dynamic
}
}
}
virtual_server 192.168.0.7 6380 {
delay_loop 3
lb_algo rr
lb_kind DR
#nat_mask 255.255.255.0
persistence_timeout 15
protocol TCP
real_server 192.168.0.2 6380 {
weight 8
notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.2 6380"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.2 6380"
misc_timeout 5
misc_dynamic
}
}
real_server 192.168.0.3 6380 {
weight 3
notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.3 6380"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.3 6380"
misc_timeout 5
misc_dynamic
}
}
real_server 192.168.0.4 6380 {
weight 3
notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.4 6380"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.4 6380"
misc_timeout 5
msic_dynamic
}
}
}
virtual_server 192.168.0.8 6379 {
delay_loop 3
lb_algo wrr
lb_kind DR
persistence_timeout 30
protocol TCP
real_server 192.168.0.2 6379 {
weight 6
notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.2 6379"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.2 6379"
misc_timeout 5
misc_dynamic
}
}
real_server 192.168.0.3 6379 {
weight 2
notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.3 6379"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.3 6379"
misc_timeout 5
misc_dynamic
}
}
real_server 192.168.0.4 6379 {
weight 2
notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.4 6379"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.4 6379"
misc_timeout 5
misc_dynamic
}
}
}
virtual_server 192.168.0.8 6380 {
delay_loop 3
lb_algo wrr
lb_kind DR
persistence_timeout 30
protocol TCP
real_server 192.168.0.2 6380 {
weight 6
notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.2 6380"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.2 6380"
misc_timeout 5
misc_dynamic
}
}
real_server 192.168.0.3 6380 {
weight 2
notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.3 6380"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.3 6380"
misc_timeout 5
misc_dynamic
}
}
real_server 192.168.0.4 6380 {
weight 2
notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.4 6380"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.4 6380"
misc_timeout 5
misc_dynamic
}
}
}
route2的keepalived配置文件! Configuration File for keepalived
global_defs {
notification_email {
接受邮箱
}
notification_email_from 发送邮箱
smtp_server 邮件服务器
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state BACKUP
interface eth1
lvs_sync_daemon_inteface eth1
virtual_router_id 100
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.0.7
}
}
vrrp_instance VI_2 {
state MASTER
interface eth1
lvs_sync_daemon_inteface eth1
virtual_router_id 101
priority 151
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.0.8
}
}
virtual_server 192.168.0.7 6379 {
delay_loop 3
lb_algo rr
lb_kind DR
persistence_timeout 15
protocol TCP
real_server 192.168.0.2 6379 {
weight 8
notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.2 6379"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.2 6379"
misc_timeout 5
misc_dynamic
}
}
real_server 192.168.0.3 6379 {
weight 3
notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.3 6379"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.3 6379"
misc_timeout 5
misc_dynamic
}
}
real_server 192.168.0.4 6379 {
weight 3
notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.4 6379"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.4 6379"
misc_timeout 5
misc_dynamic
}
}
}
virtual_server 192.168.0.7 6380 {
delay_loop 3
lb_algo rr
lb_kind DR
persistence_timeout 15
protocol TCP
real_server 192.168.0.2 6380 {
weight 8
notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.2 6380"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.2 6380"
misc_timeout 5
misc_dynamic
}
}
real_server 192.168.0.3 6380 {
weight 3
notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.3 6380"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.3 6380"
misc_timeout 5
misc_dynamic
}
}
real_server 192.168.0.4 6380 {
weight 3
notify_up "/usr/local/redis/scripts/master_config_set.py 192.168.0.4 6380"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/redischeck.py 192.168.0.4 6380"
misc_timeout 5
misc_dynamic
}
}
}
virtual_server 192.168.0.8 6379 {
delay_loop 3
lb_algo wrr
lb_kind DR
persistence_timeout 30
protocol TCP
real_server 192.168.0.2 6379 {
weight 1
notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.2 6379"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.2 6379"
misc_timeout 5
misc_dynamic
}
}
real_server 192.168.0.3 6379 {
weight 2
notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.3 6379"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.3 6379"
misc_timeout 5
misc_dynamic
}
}
real_server 192.168.0.4 6379 {
weight 2
notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.4 6379"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.4 6379"
misc_timeout 5
misc_dynamic
}
}
}
virtual_server 192.168.0.8 6380 {
delay_loop 3
lb_algo wrr
lb_kind DR
persistence_timeout 30
protocol TCP
real_server 192.168.0.2 6380 {
weight 1
notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.2 6380"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.2 6380"
misc_timeout 5
misc_dynamic
}
}
real_server 192.168.0.3 6380 {
weight 2
notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.3 6380"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.3 6380"
misc_timeout 5
misc_dynamic
}
}
real_server 192.168.0.4 6380 {
weight 2
notify_up "/usr/local/redis/scripts/slave_config_set.py 192.168.0.4 6380"
MISC_CHECK {
misc_path "/usr/local/redis/scripts/weightchange.py 192.168.0.4 6380"
misc_timeout 5
misc_dynamic
}
}
}
在keepalived使用的脚本
RunCmd.py
#!/usr/bin/python
import os;
import sys;
import time;
import fcntl;
import select;
import signal;
import commands;
import subprocess;
class RunCmd:
def __init__(self):
pass;
def Run(self,ip,port, nTimeOut = 0, nIntervalTime = 0.1):
lsCmd=['/usr/local/redis/bin/redis-cli','-h',ip,'-p',port,'ping']
oProc = subprocess.Popen(lsCmd, stdout =subprocess.PIPE, stderr = subprocess.PIPE)
istimeout=False
nStartTime = time.time()
while True:
time.sleep(nIntervalTime)
print("1:")
print(oProc.poll())
if None != oProc.poll():
break;
if (nTimeOut > 0) and (time.time() - nStartTime) > nTimeOut:
istimeout=True
break;
print("2:")
print(istimeout)
if istimeout:
print(oProc.poll())
if None == oProc.poll():
self.KillAll(oProc.pid)
print("3:")
print(istimeout)
return istimeout
def KillAll(self, nKillPid, nKillSignal = signal.SIGKILL):
nRet, strOutput = commands.getstatusoutput("kill "+str(nKillPid));#as root run
return (True, strOutput)
master_config_set.py脚本
#!/usr/bin/python
from RunCmd import RunCmd
import sys,commands
oCmd = RunCmd();
istimeout = oCmd.Run(sys.argv[1],sys.argv[2], 0.1)
if not istimeout:
cmd="/usr/local/redis/bin/redis-cli -h "+sys.argv[1]+" -p "+sys.argv[2]+" info"
str=commands.getoutput(cmd)
ismaster=str.count("role:master")
zero=0
if ismaster>zero:
t=commands.getoutput("/usr/local/redis/bin/redis-cli -h "+sys.argv[1]+" -p "+sys.argv[2]+" config set save \"\"")
print t
slave_config_set.py
#!/usr/bin/python
from RunCmd import RunCmd
import sys,commands
oCmd = RunCmd();
istimeout = oCmd.Run(sys.argv[1],sys.argv[2], 0.1)
if not istimeout:
cmd="/usr/local/redis/bin/redis-cli -h "+sys.argv[1]+" -p "+sys.argv[2]+" info"
str=commands.getoutput(cmd)
isslave=str.count("role:slave")
zero=0
if isslave>zero:
t=commands.getoutput("/usr/local/redis/bin/redis-cli -h "+sys.argv[1]+" -p "+sys.argv[2]+" config set save \"90 1 300 10 60 1000\"")
print t
redischeck.py
#!/usr/bin/python
from RunCmd import RunCmd
import sys,commands
oCmd = RunCmd();
istimeout = oCmd.Run(sys.argv[1],sys.argv[2], 0.1)
if not istimeout:
cmd="/usr/local/redis/bin/redis-cli -h "+sys.argv[1]+" -p "+sys.argv[2]+" info"
str=commands.getoutput(cmd)
ismaster=str.count("role:master")
zero=0
if ismaster>zero:
sys.exit(0)
else:
sys.exit(1)
else:
sys.exit(1)
weightchange.py
#!/usr/bin/python
from RunCmd import RunCmd
import sys,commands
oCmd = RunCmd();
istimeout = oCmd.Run(sys.argv[1],sys.argv[2], 0.1)
if not istimeout:
result=1
cmd="/usr/local/redis/bin/redis-cli -h "+sys.argv[1]+" -p "+sys.argv[2]+" ping"
strping=commands.getoutput(cmd)
zero=0
ispong=-100
ispong=strping.count("PONG")
if ispong>zero:
result=0
if result>zero:
sys.exit(1)
else:
cmdmaster="/usr/local/redis/bin/redis-cli -h "+sys.argv[1]+" -p "+sys.argv[2]+" info"
str=commands.getoutput(cmdmaster)
ismaster=-100
ismaster=str.count("role:master")
if ismaster>zero:
sys.exit(3)
else:
sys.exit(10)
else:
sys.exit(1)
redis的哨兵的配置文件sentinel.conf
# Example sentinel.conf # port <sentinel-port> # The port that this sentinel instance will run on port 26379 # sentinel monitor <master-name> <ip> <redis-port> <quorum> # # Tells Sentinel to monitor this slave, and to consider it in O_DOWN # (Objectively Down) state only if at least <quorum> sentinels agree. # # Note: master name should not include special characters or spaces. # The valid charset is A-z 0-9 and the three characters ".-_". sentinel monitor mymaster 192.168.0.2 6379 2 sentinel monitor mymaster6380 192.168.0.2 6380 2 # sentinel auth-pass <master-name> <password> # # Set the password to use to authenticate with the master and slaves. # Useful if there is a password set in the Redis instances to monitor. # # Note that the master password is also used for slaves, so it is not # possible to set a different password in masters and slaves instances # if you want to be able to monitor these instances with Sentinel. # # However you can have Redis instances without the authentication enabled # mixed with Redis instances requiring the authentication (as long as the # password set is the same for all the instances requiring the password) as # the AUTH command will have no effect in Redis instances with authentication # switched off. # # Example: # # sentinel auth-pass mymaster MySUPER--secret-0123passw0rd # sentinel down-after-milliseconds <master-name> <milliseconds> # # Number of milliseconds the master (or any attached slave or sentinel) should # be unreachable (as in, not acceptable reply to PING, continuously, for the # specified period) in order to consider it in S_DOWN state (Subjectively # Down). # # Default is 30 seconds. sentinel down-after-milliseconds mymaster 3800 sentinel down-after-milliseconds mymaster6380 3800 # sentinel can-failover <master-name> <yes|no> # # Specify if this Sentinel can start the failover for this master. sentinel can-failover mymaster yes sentinel can-failover mymaster6380 yes # sentinel parallel-syncs <master-name> <numslaves> # # How many slaves we can reconfigure to point to the new slave simultaneously # during the failover. Use a low number if you use the slaves to serve query # to avoid that all the slaves will be unreachable at about the same # time while performing the synchronization with the master. sentinel parallel-syncs mymaster 1 sentinel parallel-syncs mymaster6380 1 # sentinel failover-timeout <master-name> <milliseconds> # # Specifies the failover timeout in milliseconds. When this time has elapsed # without any progress in the failover process, it is considered concluded by # the sentinel even if not all the attached slaves were correctly configured # to replicate with the new master (however a "best effort" SLAVEOF command # is sent to all the slaves before). # # Also when 25% of this time has elapsed without any advancement, and there # is a leader switch (the sentinel did not started the failover but is now # elected as leader), the sentinel will continue the failover doing a # "takeover". # # Default is 15 minutes. sentinel failover-timeout mymaster 90000 sentinel failover-timeout mymaster6380 90000 # SCRIPTS EXECUTION # # sentinel notification-script and sentinel reconfig-script are used in order # to configure scripts that are called to notify the system administrator # or to reconfigure clients after a failover. The scripts are executed # with the following rules for error handling: # # If script exists with "1" the execution is retried later (up to a maximum # number of times currently set to 10). # # If script exists with "2" (or an higher value) the script execution is # not retried. # # If script terminates because it receives a signal the behavior is the same # as exit code 1. # # A script has a maximum running time of 60 seconds. After this limit is # reached the script is terminated with a SIGKILL and the execution retried. # NOTIFICATION SCRIPT # # sentinel notification-script <master-name> <script-path> # # Call the specified notification script for any sentienl event that is # generated in the WARNING level (for instance -sdown, -odown, and so forth). # This script should notify the system administrator via email, SMS, or any # other messaging system, that there is something wrong with the monitored # Redis systems. # # The script is called with just two arguments: the first is the event type # and the second the event description. # # The script must exist and be executable in order for sentinel to start if # this option is provided. # # Example: # # sentinel notification-script mymaster /var/redis/notify.sh # CLIENTS RECONFIGURATION SCRIPT # # sentinel client-reconfig-script <master-name> <script-path> # # When the failover starts, ends, or is aborted, a script can be called in # order to perform application-specific tasks to notify the clients that the # configuration has changed and the master is at a different address. # # The script is called in the following cases: # # Failover started (a slave is already promoted) # Failover finished (all the additional slaves already reconfigured) # Failover aborted (in that case the script was previously called when the # failover started, and now gets called again with swapped # addresses). # # The following arguments are passed to the script: # # <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port> # # <state> is "start", "end" or "abort" # <role> is either "leader" or "observer" # # The arguments from-ip, from-port, to-ip, to-port are used to communicate # the old address of the master and the new address of the elected slave # (now a master) in the case state is "start" or "end". # # For abort instead the "from" is the address of the promoted slave and # "to" is the address of the original master address, since the failover # was aborted. # # This script should be resistant to multiple invocations. # # Example: # # sentinel client-reconfig-script mymaster /var/redis/reconfig.sh
在两个route上修改/etc/sysctl.conf文件
net.ipv4.ip_forward=1#转发开启
执行sysctl -p让文件起效
有防火墙需要设置防火墙转发
