redis集群+主从部署

1：redis的高可用及主从采用keepalived以及Twemproxy ,如下图所示

　　　　

　　　　

　　　　

 

twemproxy特性：

• 支持失败节点自动删除

  • 可以设置重新连接该节点的时间
  • 可以设置连接多少次之后删除该节点
  • 该方式适合作为cache存储

• 支持设置HashTag

  • 通过HashTag可以自己设定将两个KEYhash到同一个实例上去。

• 减少与redis的直接连接数

  • 保持与redis的长连接
  • 可设置代理与后台每个redis连接的数目

• 自动分片到后端多个redis实例上

  • 多种hash算法：能够使用不同的策略和散列函数支持一致性hash。
  • 可以设置后端实例的权重

• 避免单点问题

  • 可以平行部署多个代理层.client自动选择可用的一个

• 支持redis pipelining request

       支持请求的流式与批处理，降低来回的消耗

• 支持状态监控

  • 可设置状态监控ip和端口，访问ip和端口可以得到一个json格式的状态信息串
  • 可设置监控信息刷新间隔时间

• 高吞吐量

  • 连接复用，内存复用。
  • 将多个连接请求，组成reids pipelining统一向redis请求。

     　　　　另外可以修改redis的源代码，抽取出redis中的前半部分，作为一个中间代理层。最终都是通过linux下的epoll 事件机制提高并发效率，其中nutcraker本身也是使用epoll的事件机制。并且在性能测试上的表现非常出色。

 

 

　　　　　　Twemproxy 由于其自身原理限制，有一些不足之处，如： 

• 不支持针对多个值的操作，比如取sets的子交并补等（MGET 和 DEL 除外）
• 不支持Redis的事务操作
• 出错提示还不够完善
• 也不支持select操作

2：下载twemproxy

https://github.com/twitter/twemproxy/downloads

 

tar -zxvf nutcracker-0.2.1.tar.gz

cd nutcracker

./configure --prefix=/usr/local/nutcracker

make && make install

cp conf/nutcracker.yml /etc/

 

查看下nutcracker的默认配置

cat /etc/nutcracker.yml

 

alpha:
  listen: 127.0.0.1:22121
  hash: fnv1a_64
  distribution: ketama
  auto_eject_hosts: true
  redis: true
  server_retry_timeout: 2000
  server_failure_limit: 1
  servers:
   - 127.0.0.1:6379:1

beta:
  listen: 127.0.0.1:22122
  hash: fnv1a_64
  distribution: ketama
  auto_eject_hosts: false
  timeout: 400
  redis: true
  servers:
   - 127.0.0.1:6380:1 server1
   - 127.0.0.1:6381:1 server2
   - 127.0.0.1:6382:1 server3
   - 127.0.0.1:6383:1 server4

gamma:
  listen: 127.0.0.1:22123
  hash: fnv1a_64
  distribution: ketama
  timeout: 400
  backlog: 1024
  preconnect: true
  auto_eject_hosts: true
  server_retry_timeout: 2000
  server_failure_limit: 3
  servers:
   - 127.0.0.1:11212:1
   - 127.0.0.1:11213:1

delta:
  listen: 127.0.0.1:22124
  hash: fnv1a_64
  distribution: ketama
  timeout: 100
  auto_eject_hosts: true
  server_retry_timeout: 2000
  server_failure_limit: 1
  servers:
   - 127.0.0.1:11214:1
   - 127.0.0.1:11215:1
   - 127.0.0.1:11216:1
   - 127.0.0.1:11217:1
   - 127.0.0.1:11218:1
   - 127.0.0.1:11219:1
   - 127.0.0.1:11220:1
   - 127.0.0.1:11221:1
   - 127.0.0.1:11222:1

   - 127.0.0.1:11223:1

omega:
  listen: /tmp/gamma
  hash: hsieh
  distribution: ketama
  auto_eject_hosts: false
  servers:
   - 127.0.0.1:11214:100000
   - 127.0.0.1:11215:1

 

重新配置文件,修改如下：

alpha:
  listen: 127.0.0.1:22121
  hash: fnv1a_64
  distribution: ketama
  auto_eject_hosts: true
  redis: true
  server_retry_timeout: 2000
  server_failure_limit: 1
  servers:
   - 127.0.0.1:6380:1 server1
   - 127.0.0.1:6381:1 server2
 #  - 127.0.0.1:6382:1 server3
 #  - 127.0.0.1:6383:1 server4
#beta:
#  listen: 127.0.0.1:22122
#  hash: fnv1a_64
#  distribution: ketama
#  auto_eject_hosts: false
#  timeout: 400
#  redis: true
#  servers:
#   - 127.0.0.1:6380:1 server1
#   - 127.0.0.1:6381:1 server2
#   - 127.0.0.1:6382:1 server3
#   - 127.0.0.1:6383:1 server4

 

 

 

[root@test src]# ./nutcracker -v
nutcracker: option -v requires a number
Usage: nutcracker [-?hVdDt] [-v verbosity level] [-o output file]
                  [-c conf file] [-s stats port] [-a stats addr]
                  [-i stats interval] [-p pid file] [-m mbuf size]

Options:
  -h, --help             : this help
  -V, --version          : show version and exit
  -t, --test-conf        : test configuration for syntax errors and exit
  -d, --daemonize        : run as a daemon
  -D, --describe-stats   : print stats description and exit
  -v, --verbosity=N      : set logging level (default: 5, min: 0, max: 11)
  -o, --output=S         : set logging file (default: stderr)
  -c, --conf-file=S      : set configuration file (default: conf/nutcracker.yml)
  -s, --stats-port=N     : set stats monitoring port (default: 22222)
  -a, --stats-addr=S     : set stats monitoring ip (default: 0.0.0.0)
  -i, --stats-interval=N : set stats aggregation interval in msec (default: 30000 msec)
  -p, --pid-file=S       : set pid file (default: off)
  -m, --mbuf-size=N      : set size of mbuf chunk in bytes (default: 16384 bytes)

 

 

启动服务：./nutcracker -c /etc/nutcracker.yml -d -v 6          指向文件/etc/nu... 后台启动 log等级为6

 (测试可参考：http://www.oschina.net/question/861681_135017)

3：安装redis服务

wget http://download.redis.io/releases/redis-3.0.2.tar.gz
wget http://downloads.sourceforge.net/tcl/tcl8.6.1-src.tar.gz  

tar  -zxvf redis-3.0.2.tar.gz
cd redis-3.0.2
make && make install

cd src
　　
echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf
echo 65535 > /proc/sys/net/core/somaxconn



[root@test src]# ./redis-server -h
Usage: ./redis-server [/path/to/redis.conf] [options]
       ./redis-server - (read config from stdin)
       ./redis-server -v or --version
       ./redis-server -h or --help
       ./redis-server --test-memory <megabytes>

Examples:
       ./redis-server (run the server with default conf)
       ./redis-server /etc/redis/6379.conf
       ./redis-server --port 7777
       ./redis-server --port 7777 --slaveof 127.0.0.1 8888
       ./redis-server /etc/myredis.conf --loglevel verbose

Sentinel mode:
       ./redis-server /etc/sentinel.conf --sentinel


根据上面的配置启动redis端口：

./redis-server /usr/local/src/redis-3.0.2/master-server/6379/redis.conf
./redis-server /usr/local/src/redis-3.0.2/master-server/6380/redis.conf   
./redis-server /usr/local/src/redis-3.0.2/master-server/6381/redis.conf   
./redis-server /usr/local/src/redis-3.0.2/master-server/6382/redis.conf      

测试性能：
redis-benchmark -h 192.168.16.75 -p 22121 -c 100 -t set -d 100 -l –q

[root@test src]# ./redis-benchmark -h 192.168.16.75 -p 22122
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
All clients disconnected... aborting.
[root@test src]# ./redis-benchmark -h 192.168.16.75 -p 22121
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
Writing to socket: Connection refused
All clients disconnected... aborting.


[root@test src]# ./redis-benchmark -h 192.168.16.75 -p 6380 -c 100 -t set -d 100 -l –q     
====== –q ======
  100000 requests completed in 1.15 seconds

可见nu无法少量并发，（为什么？？？）

------------------------------------------------根据redis自带集群进行部署：

port 7000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

yum -y install ruby
yum -y install rubygems
gem install redis --version 3.0.0  
由于源的原因，可能下载失败，就手动下载下来安装  
download地址:http://rubygems.org/gems/redis/versions/3.0.0  
gem install -l /data/soft/redis-3.0.0.gem  

./redis-trib.rb:24:in `require': no such file to load -- rubygems (LoadError)
        from ./redis-trib.rb:24

/usr/local/src/redis-3.0.2/src/redis-server /usr/local/src/redis-3.0.2/cluster-test/6380/redis6380.conf
/usr/local/src/redis-3.0.2/src/redis-server /usr/local/src/redis-3.0.2/cluster-test/6381/redis6381.conf

/usr/local/src/redis-3.0.2/src/redis-server /usr/local/src/redis-3.0.2/cluster-test/6382/redis6382.conf
/usr/local/src/redis-3.0.2/src/redis-server /usr/local/src/redis-3.0.2/cluster-test/6383/redis6383.conf

/usr/local/src/redis-3.0.2/src/redis-server /usr/local/src/redis-3.0.2/cluster-test/6384/redis6384.conf
/usr/local/src/redis-3.0.2/src/redis-server /usr/local/src/redis-3.0.2/cluster-test/6385/redis6385.conf

如果直接到src目录下启动，cluster的node.conf appendonly dump都会生成在同一目录下，造成其他的实例无法启动


[root@test src]# ./redis-trib.rb create  --replicas 1  127.0.0.1:6380  127.0.0.1:6381 127.0.0.1:6382 127.0.0.1:6383 127.0.0.1:6384 127.0.0.1:6385
>>> Creating cluster
Connecting to node 127.0.0.1:6380: OK
Connecting to node 127.0.0.1:6381: OK
Connecting to node 127.0.0.1:6382: OK
Connecting to node 127.0.0.1:6383: OK
Connecting to node 127.0.0.1:6384: OK
Connecting to node 127.0.0.1:6385: OK
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:6380
127.0.0.1:6381
127.0.0.1:6382
Adding replica 127.0.0.1:6383 to 127.0.0.1:6380
Adding replica 127.0.0.1:6384 to 127.0.0.1:6381
Adding replica 127.0.0.1:6385 to 127.0.0.1:6382
M: be241eb8ed58d1f51fee67e15854d4c52c56e5f8 127.0.0.1:6380
   slots:0-5460 (5461 slots) master
M: ea812a18dfd36a1c063c050aee90d2f1d93d411d 127.0.0.1:6381
   slots:5461-10922 (5462 slots) master
M: bf0fa514d1ae83db6c3a2fddcff8412647dc04fc 127.0.0.1:6382
   slots:10923-16383 (5461 slots) master
S: a7e32ea6bffdf655ac1b4bd7b425db9909c0f8cb 127.0.0.1:6383
   replicates be241eb8ed58d1f51fee67e15854d4c52c56e5f8
S: 18516163fe5988d6d4687d4ccad4962629c023a6 127.0.0.1:6384
   replicates ea812a18dfd36a1c063c050aee90d2f1d93d411d
S: 991761a527e2f3de09733440a72f1abe2d6673a0 127.0.0.1:6385
   replicates bf0fa514d1ae83db6c3a2fddcff8412647dc04fc
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join....
>>> Performing Cluster Check (using node 127.0.0.1:6380)
M: be241eb8ed58d1f51fee67e15854d4c52c56e5f8 127.0.0.1:6380
   slots:0-5460 (5461 slots) master
M: ea812a18dfd36a1c063c050aee90d2f1d93d411d 127.0.0.1:6381
   slots:5461-10922 (5462 slots) master
M: bf0fa514d1ae83db6c3a2fddcff8412647dc04fc 127.0.0.1:6382
   slots:10923-16383 (5461 slots) master
M: a7e32ea6bffdf655ac1b4bd7b425db9909c0f8cb 127.0.0.1:6383
   slots: (0 slots) master
   replicates be241eb8ed58d1f51fee67e15854d4c52c56e5f8
M: 18516163fe5988d6d4687d4ccad4962629c023a6 127.0.0.1:6384
   slots: (0 slots) master
   replicates ea812a18dfd36a1c063c050aee90d2f1d93d411d
M: 991761a527e2f3de09733440a72f1abe2d6673a0 127.0.0.1:6385
   slots: (0 slots) master
   replicates bf0fa514d1ae83db6c3a2fddcff8412647dc04fc
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.


[root@test src]# ./redis-cli -h
redis-cli 3.0.2

Usage: redis-cli [OPTIONS] [cmd [arg [arg ...]]]
  -h <hostname>      Server hostname (default: 127.0.0.1).
  -p <port>          Server port (default: 6379).
  -s <socket>        Server socket (overrides hostname and port).
  -a <password>      Password to use when connecting to the server.
  -r <repeat>        Execute specified command N times.
  -i <interval>      When -r is used, waits <interval> seconds per command.
                     It is possible to specify sub-second times like -i 0.1.
  -n <db>            Database number.
  -x                 Read last argument from STDIN.
  -d <delimiter>     Multi-bulk delimiter in for raw formatting (default: \n).
  -c                 Enable cluster mode (follow -ASK and -MOVED redirections).
  --raw              Use raw formatting for replies (default when STDOUT is
                     not a tty).
  --no-raw           Force formatted output even when STDOUT is not a tty.
  --csv              Output in CSV format.
  --stat             Print rolling stats about server: mem, clients, ...
  --latency          Enter a special mode continuously sampling latency.
  --latency-history  Like --latency but tracking latency changes over time.
                     Default time interval is 15 sec. Change it using -i.
  --latency-dist     Shows latency as a spectrum, requires xterm 256 colors.
                     Default time interval is 1 sec. Change it using -i.
  --lru-test <keys>  Simulate a cache workload with an 80-20 distribution.
  --slave            Simulate a slave showing commands received from the master.
  --rdb <filename>   Transfer an RDB dump from remote server to local file.
  --pipe             Transfer raw Redis protocol from stdin to server.
  --pipe-timeout <n> In --pipe mode, abort with error if after sending all data.
                     no reply is received within <n> seconds.
                     Default timeout: 30. Use 0 to wait forever.
  --bigkeys          Sample Redis keys looking for big keys.
  --scan             List all keys using the SCAN command.
  --pattern <pat>    Useful with --scan to specify a SCAN pattern.
  --intrinsic-latency <sec> Run a test to measure intrinsic system latency.
                     The test will run for the specified amount of seconds.
  --eval <file>      Send an EVAL command using the Lua script at <file>.
  --help             Output this help and exit.
  --version          Output version and exit.

Examples:
  cat /etc/passwd | redis-cli -x set mypasswd
  redis-cli get mypasswd
  redis-cli -r 100 lpush mylist x
  redis-cli -r 100 -i 1 info | grep used_memory_human:
  redis-cli --eval myscript.lua key1 key2 , arg1 arg2 arg3
  redis-cli --scan --pattern '*:12345*'

  (Note: when using --eval the comma separates KEYS[] from ARGV[] items)

When no command is given, redis-cli starts in interactive mode.
Type "help" in interactive mode for information on available commands.

[root@test src]# ./redis-cli -c -p 6385
127.0.0.1:6385> set foo bar
-> Redirected to slot [12182] located at 127.0.0.1:6382
OK
127.0.0.1:6382> set foo bar
OK
127.0.0.1:6382> set foor bar
-> Redirected to slot [3280] located at 127.0.0.1:6380
OK
127.0.0.1:6380> set foo bar
-> Redirected to slot [12182] located at 127.0.0.1:6382
OK
127.0.0.1:6382> 


实现主从redis故障转移，采用sentinel模式启动，另外需要写两个脚本，实现配置文件的转移


参考文档：http://blog.sina.com.cn/s/blog_48c95a190101di13.html
　　　　　http://www.bkjia.com/sybase/1010631.html
　　　　　http://www.cnblogs.com/haoxinyue/p/redis.html