Redis Cluster集群搭建与配置
Redis Cluster是一种服务器sharding分片技术，关于Redis的集群方案应该怎么做，请参考我的另一篇博客http://www.cnblogs.com/xckk/p/6134655.html
本文主要介绍Redis Cluster集群的工作原理，详细讲解了Redis Cluster集群如何搭建与配置。
一、redis安装
redis官网下载后是源码包，需要make安装。
1、解压redis-3.2.5.tar.gz
2、cd redis-3.2.5
3、make
4、cd src/
5、make install
安装完成，src包下面新增redis-server、redis-cli等这些可执行脚本，表示安装完成。
二、启动实例集群配置
搭建Redis Cluster，建立6个redis实例，3主3从，端口号分别为7000-7005
创建以6个端口命名的子目录，之后我们将在6个子目录上都开启Redis实例
cd redis-3.2.5 mkdir cluster-test cd cluster-test mkdir 7000 7001 7002 7003 7004 7005
Redis实例配置
在7000到7005每一个目录中都创建redis.conf文件，配置项如下：
cd redis-3.2.5 mkdir cluster-test cd cluster-test mkdir 7000 7001 7002 7003 7004 7005
各配置描述如下：
|
配置项redis.conf |
示例 |
备注 |
| port | 7000 | 端口号 |
| cluster-enabled | yes |
yes，表示支持Redis Cluster |
| cluster-config-file | nodes-7000.conf |
Redis Cluster记录的启动信息文件，文件由cluster自动生成，不需要用户编辑 |
| cluster-node-timeout | 5000 | 毫秒单位，失效时间 |
| appendonly | yes | 支持appendonly持久化方式 |
| protected-mode | no | 默认是打开的，此处关闭。因为jedis访问时，如果是保护模式，影响访问 |
| bind | 192.168.121.130 | 机器IP，此处最好写IP，不要写localhost或127.0.0.1，可能影响jedis客户端访问 |
Redis实例启动
将第一步安装生成的redis-server拷贝到cluster-test目录，分别后台启动6个redis实例
nohup ./redis-server 7000/redis.conf & nohup ./redis-server 7001/redis.conf & nohup ./redis-server 7002/redis.conf & nohup ./redis-server 7003/redis.conf & nohup ./redis-server 7004/redis.conf & nohup ./redis-server 7005/redis.conf &
启动完后，cluster-test目录下生成了nodes-7000.conf等启动文件信息。
查看进程启动成功：[root@centos2 cluster-test]# ps -ef|grep redis
三、创建集群
6个redis实例已经创建，它们目前是相互独立的，没有关联。接下来是创建Redis Cluster集群，将6个实例组成一个集群。
redis-trib命令创建集群
src目录下make生成的redis-trib命令，可用于Redis Cluster集群创建的一些脚本启动，它是一个ruby程序。因此我们需要配置ruby环境
安装ruby
yum install ruby
yum install ruby-irb
具体ruby的安装可参考：http://jingyan.baidu.com/article/b7001fe173fe9a0e7382dd57.html
安装Gem
Gem是一个管理Ruby库和程序的标准包，它通过RubyGem（如 http://rubygems.org/ ）源来查找、安装、升级和卸载软件包，非常的便捷。
配置了ruby环境后，还需要安装redis整个包才能够运行redis-trib。安装过程中，需要先安装gem库，不然会出现gem command not found错误。
[root@centos2 src]# gem install redis -bash: gem: command not found
安装gem过程中，可能出现以下问题
yum install rubygems ERROR: Could not find a valid gem 'redis' (>= 0) in any repository
问题原因：无法连接gem服务器，原因是该gem服务器被墙了。按如下步骤手动下载安装。
wget https://rubygems.global.ssl.fastly.net/gems/redis-3.2.1.gem gem install -l ./redis-3.2.1.gem
安装完后，src包下就会生成redis-trib.rb文件
启动命令
./redis-trib.rb create --replicas 1 192.168.121.130:7000 192.168.121.130:7001 192.168.121.130:7002 192.168.121.130:7003 192.168.121.130:7004 192.168.121.130:7005
启动后，各Redis实例之间分别会新增一个端口用于通信，通信端口是Redis实例端口+1W，分别为17000，17001，17002，17003，17004，17005
命令的解释如下：
1、给定的命令将会被ruby程序翻译为create，这表示我们想要创建一个新的集群。
2、选项--replicas 1意思是为每一个master节点创建一个slave节点。
3、其他参数表示redis实例的地址及端口，以空格为间隔。
启动日志分析
出现[OK] All 16384 slots covered.表示启动成功。从启动日志可以看出，master是7000,7001,7002，对应的slave是7003,7004,7005
[root@centos1 src]# ./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 >>> Creating cluster >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 Adding replica 127.0.0.1:7003 to 127.0.0.1:7000 Adding replica 127.0.0.1:7004 to 127.0.0.1:7001 Adding replica 127.0.0.1:7005 to 127.0.0.1:7002 M: 434e5ee5cf198626e32d71a4aee27bc4058b4e45 127.0.0.1:7000 slots:0-5460 (5461 slots) master M: 048a0c9631c87e5ecc97a4ce5834d935f2f938b6 127.0.0.1:7001 slots:5461-10922 (5462 slots) master M: 04ae4184b2853afb8122d15b5b2efa471d4ca251 127.0.0.1:7002 slots:10923-16383 (5461 slots) master S: 499b0bfa9274425bfcb87f7aa3da76456c3397da 127.0.0.1:7003 replicates 434e5ee5cf198626e32d71a4aee27bc4058b4e45 S: 7f92536844f1698b5776c2f0823f1822f0bb88d7 127.0.0.1:7004 replicates 048a0c9631c87e5ecc97a4ce5834d935f2f938b6 S: 411bb5ea8c07d872033f7c473c35fe38416052ce 127.0.0.1:7005 replicates 04ae4184b2853afb8122d15b5b2efa471d4ca251 Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join... >>> Performing Cluster Check (using node 127.0.0.1:7000) M: 434e5ee5cf198626e32d71a4aee27bc4058b4e45 127.0.0.1:7000 slots:0-5460 (5461 slots) master 1 additional replica(s) S: 411bb5ea8c07d872033f7c473c35fe38416052ce 127.0.0.1:7005 slots: (0 slots) slave replicates 04ae4184b2853afb8122d15b5b2efa471d4ca251 S: 7f92536844f1698b5776c2f0823f1822f0bb88d7 127.0.0.1:7004 slots: (0 slots) slave replicates 048a0c9631c87e5ecc97a4ce5834d935f2f938b6 M: 048a0c9631c87e5ecc97a4ce5834d935f2f938b6 127.0.0.1:7001 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: 499b0bfa9274425bfcb87f7aa3da76456c3397da 127.0.0.1:7003 slots: (0 slots) slave replicates 434e5ee5cf198626e32d71a4aee27bc4058b4e45 M: 04ae4184b2853afb8122d15b5b2efa471d4ca251 127.0.0.1:7002 slots:10923-16383 (5461 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
集群启动状态检测
我们可以通过./redis-trib.rb check 192.168.121.130:7000来检测Redis Cluster的启动状态。
[root@centos2 src]# ./redis-trib.rb check 192.168.121.130:7000 >>> Performing Cluster Check (using node 192.168.121.130:7000) M: dca47b274799927fd15aa4e6312b94752418f0c0 192.168.121.130:7000 slots:0-5460 (5461 slots) master 1 additional replica(s) M: fe1082655e832bd2afb7d4bfabfec3ce2354868b 192.168.121.130:7002 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 7fe93723c48b7ea33bbf21f12e97d18bdd0221ea 192.168.121.130:7003 slots: (0 slots) slave replicates dca47b274799927fd15aa4e6312b94752418f0c0 M: 59c926f7bb1e24e293d379507742c52b6dd08cc2 192.168.121.130:7001 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: a554424a9969af2b55f4e11f9eedf1bc3bce9000 192.168.121.130:7004 slots: (0 slots) slave replicates 59c926f7bb1e24e293d379507742c52b6dd08cc2 S: 065fb213e9db64627c27af93500420351b80ac34 192.168.121.130:7005 slots: (0 slots) slave replicates fe1082655e832bd2afb7d4bfabfec3ce2354868b [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
登录集群
通过命令./redis-cli -c -h 192.168.121.129 -p 7000用来登录
重定向
从以下命令可以看出，我们登录的是192.168.121.130机器的7000端口Redis实例，set foo 123，foo经过hash后落到了12182槽，12182槽存储在192.168.121.130机器的7002端口实例，因此日志显示重定向到7002端口的实例。
[root@centos2 src]# ./redis-cli -c -h 192.168.121.130 -p 7000 192.168.121.130:7000> set foo 123 -> Redirected to slot [12182] located at 192.168.121.130:7002 OK 192.168.121.130:7002> get foo "123" 192.168.121.130:7002>
redis cluster的主从同步采用异步复制（提高性能），因此会存在部分写内容丢失的情况
四、创建集群-通过create-cluster脚本
通过redis-trib.rb命令创建集群涉及命令较多，稍显复杂。好在redis提供了create-cluster脚本utils/create-cluster/create-cluster来创建集群，方便很多。
create-cluster命令介绍
从create-cluster源码看，其实现原理也是利用了redis-trib.rb命令。该脚本默认创建3个master节点，3个salve节点。
1、create-cluster start
创建6个redis实例，默认端口号是30001-30006，如果想改端口号,在create-cluster里面更改PORT值即可。
如果想要端口号是7000-7005，设置PORT=6999即可
2、create-cluster create
创建集群，3个master,3个slave
3、create-cluster stop
关闭Redis Cluster集群和Redis实例
4、create-cluster clean
清除所在Redis实例的data, logs, config文件
使用方法
cd /home/redis-3.2/utils/create-cluster --进入目录
./create-cluster clean ---先清除Redis实例的文件
./create-cluster start ---启动Redis实例
./create-cluster create ---创建Redis Cluster集群
启动完后，使用redis-cli命令登录，即可正常访问。
[root@centos1 create-cluster]# redis-cli -c -p 7000 127.0.0.1:7000> set foo hello -> Redirected to slot [12182] located at 127.0.0.1:7002 OK 127.0.0.1:7002> set hello world -> Redirected to slot [866] located at 127.0.0.1:7000 OK 127.0.0.1:7000> get foo -> Redirected to slot [12182] located at 127.0.0.1:7002 "hello" 127.0.0.1:7002> get hello -> Redirected to slot [866] located at 127.0.0.1:7000 "world" 127.0.0.1:7000>
五、Jedis测试程序
Jedis是Redis官方首选的 Java 客户端开发包。我们通过Junit进行单元测试
maven添加jedis和junit依赖包
<dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> <scope>test</scope> </dependency> <!-- https://mvnrepository.com/artifact/redis.clients/jedis --> <dependency> <groupId>redis.clients</groupId> <artifactId>jedis</artifactId> <version>2.8.1</version> </dependency> </dependencies>
java代码
package com.hk.test.helloWorld; import java.util.HashSet; import java.util.List; import java.util.Set; import org.apache.commons.pool2.impl.GenericObjectPoolConfig; import org.junit.Before; import org.junit.Test; import redis.clients.jedis.HostAndPort; import redis.clients.jedis.JedisCluster; public class TestJedis { JedisCluster jc = null; @Before public void before() { Set<HostAndPort> jedisClusterNodes = new HashSet<HostAndPort>(); // Jedis Cluster will attempt to discover cluster nodes automatically jedisClusterNodes.add(new HostAndPort("centos2", 7000)); GenericObjectPoolConfig config; jc = new JedisCluster(jedisClusterNodes); } @Test public void testFoo() { for (int i = 0; i <= 3000; i++) { String key = "foo" + i; String value = String.valueOf(i); jc.set(key, value); } } @Test public void test_incr() { String key = "page_view"; jc.del(key); jc.incr(key); String result = jc.get(key); System.out.println(result); } @Test public void test_setAndGetStringVal() { String key = "foo"; String value = "bar"; jc.set(key, value); String result = jc.get(key); System.out.println(result); } @Test public void test_setAndGetStringVal_and_set_expire() throws InterruptedException { String key = "hello"; String value = "world"; int seconds = 3; jc.setex(key, seconds, value); String result = jc.get(key); System.out.println(result); Thread.sleep(seconds * 1000); result = jc.get(key); System.out.println(result); } @Test public void test_setAndGetHashVal() { String key = "website"; String field = "google"; String value = "google.com"; jc.del(key); jc.hset(key, field, value); String result = jc.hget(key, field); System.out.println(result); } @Test public void test_setAndGetListVal() { String key = "mylist"; jc.del(key); String[] vals = { "a", "b", "c" }; jc.rpush(key, vals); List<String> result = jc.lrange(key, 0, -1); System.out.println(result); } @Test public void test_setAndGetSetVal() { String key = "language"; jc.del(key); String[] members = { "java", "ruby", "python" }; jc.sadd(key, members); Set<String> result = jc.smembers(key); System.out.println(result); } }
六、常见问题
一、Could not connect to Redis at 127.0.0.1:30001 Connection refused
原因：create-cluster脚本默认设定的端口从30001开始，由于我设置的redis端口是7000-7005，当时启动cluster时用的redis-trib，因此端口号不同，
./redis-trib.rb create --replicas 1 192.168.121.130:7000 192.168.121.130:7001 192.168.121.130:7002 192.168.121.130:7003 192.168.121.130:7004 192.168.121.130:7005
解决方法：
1、create-cluster里面将PORT由30001改为6999。
2、关闭Redis实例进程
可以通过netstat -apn|grep redis或者ps -ef|grep redis查找进程号,kill -9 ｛进程号｝进行关闭。
3、create-cluster stop
4、create-cluster clean 清除data,log,conf文件
5、create-cluster start
6、create-cluster create
二、redis.clients.jedis.exceptions.JedisDataException: DENIED Redis is running in protected mode because protected mode is enabled
原因：Redis实例启动时默认设置了保护模式
解决方法：需要在启动时增加配置项 protected-mode no
1、如果是redis-trib.rb启动，启动配置文件内增加 protected-mode no
port 7000 cluster-enabled yes cluster-config-file nodes-7000.conf cluster-node-timeout 5000 appendonly yes protected-mode no bind 192.168.121.129
2、如果是create-cluster启动，在create-cluster脚本里start分支中增加protected-mode no
if [ "$1" == "start" ] then while [ $((PORT < ENDPORT)) != "0" ]; do PORT=$((PORT+1)) echo "Starting $PORT" ../../src/redis-server --port $PORT --cluster-enabled yes --cluster-config-file nodes-${PORT}.conf --cluster-node-timeout $TIMEOUT --appendonly yes --bind 192.168.121.130 --protected-mode no --appendfilename appendonly-${PORT}.aof --dbfilename dump-${PORT}.rdb --logfile ${PORT}.log --daemonize yes done exit 0 fi
三、Jedis访问时，redis.clients.jedis.exceptions.JedisClusterMaxRedirectionsException: Too many Cluster redirections?
若用JedisCluster操作集群报以上错误，就是有Redis节点通讯被拒绝，最好是将redis.conf文件中的bind属性设置成redis节点所在服务器的物理ip地址，而不是localhost
1、更改node节点配置文件绑定地址
2、重启redis节点
3、重新create cluster
./redis-trib.rb create --replicas 192.168.121.130:7000 192.168.121.130:7001 192.168.121.130:7002 192.168.121.130:7003192.168.121.130:7004 192.168.121.130:7005
四、[ERR] Node 172.168.63.202:7001 is not empty. Either the nodealready knows other nodes (check with CLUSTER NODES) or contains some
解决方法：（一般1，2步即可解决）
1)、将需要新增的节点下aof、rdb等本地备份文件删除；
2)、同时将新Node的集群配置文件删除,即：删除你redis.conf里面cluster-config-file所在的文件；
3)、再次添加新节点如果还是报错，则登录新Node,./redis-cli–h x –p对数据库进行清除：
172.168.63.201:7001> flushdb #清空当前数据库
4)、删除后再执行第2,3步，再./create-cluster start即可
五、ERR Invalid node address specified
./redis-trib.rb create --replicas 1 centos2:7000 centos2:7001 centos2:7002 centos2:7003 centos2:7004 centos2:7005
报" ERR Invalid node address specified"
由于Redis-trib.rb 对域名或主机名支持不好,故在创建集群的时候要使用ip:port的方式
./redis-trib.rb create --replicas 1 192.168.121.130:7000 192.168.121.130:7001 192.168.121.130:7002 192.168.121.130:7003 192.168.121.130:7004 192.168.121.130:7005
六、ERR Slot 4648 is already busy (Redis::CommandError)
这是由于之前创建集群没有成功,需要
1、将nodes.conf和dir里面的文件全部删除(注意不要删除了redis.conf)
2、将redis实例关闭后重新启动
3、再执行/redis-trib.rb create 。。。脚本
Sorry, the cluster configuration file nodes.conf is already used by a different Redis Cluster node. Please make sure that different nodes use different cluster configuration files.
七、[ERR] Node 127.0.0.1:7000 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
解决方法：先stop和clean，再start和create
./create-cluster stop
./create-cluster clean
./create-cluster start
./create-cluster create
秀才坤坤出品
转载请注明来源http://www.cnblogs.com/xckk/p/6144447.html
