Redis集群搭建的几种方式

前面使用了哨兵集群搭建单个分片高可用的结构，实现了主从复制和故障转移，但是没有实现分布式。接下来使用Redis集群搭建，实现哨兵集群没有实现的分布式高可用。

redis-cluster搭建需求

下面准备6个redis节点，搭建如图所示的三主三从分布式redis集群，实现两两互联，以master8001为例，它可以连接另外两个主节点，以及三个从节点。搭建过程可以使用原生redis cluster命令（使用方便），也可以使用ruby自带的脚本（需要先安装ruby，还有安装redis和ruby的接口，比较麻烦）。

ruby脚本辅助搭建

使用ruby脚本辅助安装，即使用redis根目录/src/redis-trib.rb脚本来完成，需要先安装ruby。

-rwxrwxr-x. 1 root root   60852 Sep 21  2017 redis-trib.rb

（1）安装ruby，这里解压了压缩包，进行编译和安装。但是安装完只是最基本的一步，后面还有坑，比较麻烦。

# 解压
[root@node01 /home/software]#  tar -zxvf  ruby-2.3.1.tar.gz
# 检查
[root@node01 /home/software/ruby-2.3.1]#  . /configure
# 编译，安装
[root@node01 /home/software/ruby-2.3.1]# make && make install
# 省略具体安装日志，安装完成后使用ruby -v如果能查看到版本号，就安装ok
[root@node01 /home/software/ruby-2.3.1]#  ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_4-linux]

（2）先准备一个redis-cluster.conf启动加载文件，基本配置参考前面博文https://www.cnblogs.com/youngchaolin/p/11983705.html#_label0 ，主要是bind设置不绑定ip、protected mode设置为no、daemonize设置为yes、其他还有pid、logfile、rdb等，保持原设置不变。

（3）配置appendonly为yes，启动aof持久化方式，这是除了rdb外redis提供的第二种持久化方式。

 589 # AOF and RDB persistence can be enabled at the same time without problems.
 590 # If the AOF is enabled on startup Redis will load the AOF, that is the file
 591 # with the better durability guarantees.
 592 #
 593 # Please check http://redis.io/topics/persistence for more information.
 594 # 配置第二种持久化方式aof
 595 appendonly yes
 596
 597 # The name of the append only file (default: "appendonly.aof")
 598 # 标识aof持久化文件名，以端口号区分
 599 appendfilename "appendonly6379.aof"

其中rdb和aof两种持久化的主要区别为：

a.rdb保存的是具体的key-value数据，如name-messi，aof保存的是操作记录，如set name messi。

b.因为aof保存的数据更全，redis启动默认加载的是aof，rdb启动可以单独开启。

c.rdb持久化频率是按照持久化策略来的，容易造成数据丢失，而aof是每秒保存一次数据，数据不容易丢失。

d.如果对数据的可靠性要求高，使用aof持久化，如果需要一定的数据恢复能力，但是又不需要很高的可靠性，就选择rdb。

（4）开启集群模式，如果不开启无法计算槽道号。

 719 # Normal Redis instances can't be part of a Redis Cluster; only nodes that are
 720 # started as cluster nodes can. In order to start a Redis instance as a
 721 # cluster node enable the cluster support uncommenting the following:
 722 # 需设置yes，否则无法计算槽道号，无法创建集群
 723 cluster-enabled yes

（5）需要开启集群节点状态记录文件，这个文件会自动更新，每个redis节点都需要一个这样的文件。

 725 # Every cluster node has a cluster configuration file. This file is not
 726 # intended to be edited by hand. It is created and updated by Redis nodes.
 727 # Every Redis Cluster node requires a different cluster configuration file.
 728 # Make sure that instances running in the same system do not have
 729 # overlapping cluster configuration file names.
 730 # 开启集群节点状态记录文件
 731 cluster-config-file nodes-6379.conf

（6）将上面的配置文件保存，接下来复制到6个不同的文件夹下，其中文件夹的名字就是redis节点的端口号，根据上图准备8000、8001、8002、8003、8004和8005六个文件夹。复制完成后，修改对应端口号，如8000文件夹的配置文件，将里面全是6379的部分替换为8000，其他类似修改即可。

# 创建目录
[root@node01 /home/software/redis-3.2.11]# mkdir 8000 8001 8002 8003 8004 8005
# 复制文件
[root@node01 /home/software/redis-3.2.11]# cp redis.conf 8000/
[root@node01 /home/software/redis-3.2.11]# cp redis.conf 8001/
[root@node01 /home/software/redis-3.2.11]# cp redis.conf 8002/
[root@node01 /home/software/redis-3.2.11]# cp redis.conf 8003/
[root@node01 /home/software/redis-3.2.11]# cp redis.conf 8004/
[root@node01 /home/software/redis-3.2.11]# cp redis.conf 8005/
# 递归查看是否copy成功
[root@node01 /home/software/redis-3.2.11]# ls -R 800*
8000:
redis.conf

8001:
redis.conf

8002:
redis.conf

8003:
redis.conf

8004:
redis.conf

8005:
redis.conf
# 修改端口略

（7）启动各个节点，发现均是cluster的方式启动，这是创建集群的基础，另外查看集群状态为fail，是因为槽道还没分配的原因，且每个节点cluster nodes查看集群节点信息都只能找到自己一个节点，目前两两互联还没有建立。

# 加载修改后的配置文件启动各个节点
[root@node01 /home/software/redis-3.2.11]# redis-server 8000/redis.conf
[root@node01 /home/software/redis-3.2.11]# redis-server 8001/redis.conf
[root@node01 /home/software/redis-3.2.11]# redis-server 8002/redis.conf
[root@node01 /home/software/redis-3.2.11]# redis-server 8003/redis.conf
[root@node01 /home/software/redis-3.2.11]# redis-server 8004/redis.conf
[root@node01 /home/software/redis-3.2.11]# redis-server 8005/redis.conf
# 查看是否以cluster方式启动，ok
[root@node01 /home/software/redis-3.2.11]# ps -ef|grep redis
root     16028     1  0 15:43 ?        00:00:00 redis-server *:8000 [cluster]
root     16033     1  0 15:43 ?        00:00:00 redis-server *:8001 [cluster]
root     16037     1  0 15:43 ?        00:00:00 redis-server *:8002 [cluster]
root     16041     1  0 15:43 ?        00:00:00 redis-server *:8003 [cluster]
root     16045     1  0 15:43 ?        00:00:00 redis-server *:8004 [cluster]
root     16049     1  0 15:43 ?        00:00:00 redis-server *:8005 [cluster]
root     16053  1409  0 15:43 pts/0    00:00:00 grep redis
# 登录一个节点，查看集群信息
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000
127.0.0.1:8000> cluster info
cluster_state:fail # 只要有一个槽道没分配，就是fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1 # 只能发现自己一个节点，两两互联暂未建立
cluster_size:0
cluster_current_epoch:0 # 集群的纪元
cluster_my_epoch:0
cluster_stats_messages_sent:0 # 两两互联发送数据量
cluster_stats_messages_received:0 # 两两互联接受数据量
# 查看集群节点，发现只有自己一个
127.0.0.1:8000> cluster nodes
dc0e2e16888426089de31466ff398679eec81b86 :8000 myself,master - 0 0 0 connected

（8）启动src/redis-trib.rb命令后想查看帮助信息发现报错。这是正常的，这意味着踏入了一个大坑，接下来的需要填坑了，参考文末博文，完成了报错解决。

# 报错
/usr/local/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- redis (LoadError)
    from /usr/local/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from src/redis-trib.rb:25:in `<main>'

（9）上面报错是需要redis的gem包来安装ruby和redis的接口，先安装gem包，网址https://rubygems.org/gems/redis/，选择对应的版本下载，我这里是选择3.2.1版本，下载好后上传到了redis根目录/src目录下了，具体位置放哪不清楚，放src下是可以验证通过的，另外查看博文放根目录也是可以的。

-rw-r--r--. 1 root root   73728 Dec 12 16:51 redis-3.2.1.gem

（10）准备使用gem install redis命令安装接口，发现报错，提示需要安装zlib，网址http://www.zlib.net，本次选择了1.2.11版本。

ERROR:  Loading command: install (LoadError)
    cannot load such file -- zlib
ERROR:  While executing gem ... (NoMethodError)
    undefined method `invoke_with_build_args' for nil:NilClass

（11）tar -zxvf解压zlib压缩包后，进入解压目录，先指定安装目录，后编译安装zlib。

# --prefix指定安装目录/usr/local/zlib
[root@node01 /home/software/zlib-1.2.11]# ./configure --prefix=/usr/local/zlib
Checking for gcc...
Checking for shared library support...
Building shared library libz.so.1.2.11 with gcc.
Checking for size_t... Yes.
Checking for off64_t... Yes.
Checking for fseeko... Yes.
Checking for strerror... Yes.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Checking for attribute(visibility) support... Yes.
You have new mail in /var/spool/mail/root
# 编译后安装
[root@node01 /home/software/zlib-1.2.11]# make && make install
...省略 安装成功
chmod 644 /usr/local/zlib/include/zlib.h /usr/local/zlib/include/zconf.h

（12）还需要进入ruby根目录/ext/zlib，先使用ruby extconf.rb命令，执行完后会在当前目录下生成Makefile文件。

[root@node01 /home/software/ruby-2.3.1/ext/zlib]# ll
total 892
-rw-r--r--. 1 root root    146 Apr 16  2013 depend
-rw-r--r--. 1 root root   1447 Dec 16  2015 extconf.rb
# 使用ruby extconf.rb命令后会生成一个Makefile文件
-rw-r--r--. 1 root root   7656 Dec 12 17:14 Makefile
-rw-r--r--. 1 root root   7090 Dec 12 17:14 mkmf.log
-rw-r--r--. 1 root root 124168 Dec 19  2015 zlib.c
-rw-r--r--. 1 root root 418576 Dec 12 17:18 zlib.o
-rwxr-xr-x. 1 root root 336032 Dec 12 17:18 zlib.so
# 安装时需添加配置信息，指定zlib目录下的include和lib，zlib目录就是前面编译指定安装的目录
[root@node01 /home/software/ruby-2.3.1/ext/zlib]# ruby extconf.rb --with-zlib-include=/usr/local/zlib/include/ --with-zlib-lib=/usr/local/zlib/lib

# 省略，编译安装
[root@node01 /home/software/ruby-2.3.1/ext/zlib]# make && make install

接下来编译，本次编译顺利通过了，如果没有通过可能会出现如下报错，还需要修改Makefile内容。

make: *** No rule to make target `/include/ruby.h', needed by `zlib.o'.  Stop

修改Makefile内容，将$(top_srcdir)整个换成绝对路径，我这里没有报错，因此没有修改。

zlib.o: $(top_srcdir)/include/ruby.h 
修改成：zlib.o: ../../include/ruby.h

最后编译通过会提示如下内容。

[root@node01 /home/software/ruby-2.3.1/ext/zlib]# make && make install
compiling zlib.c
linking shared-object zlib.so
/usr/bin/install -c -m 0755 zlib.so /usr/local/lib/ruby/site_ruby/2.4.0/x86_64-linux

（13）继续安装openssl，如果不安装切换到redis目录使用gem install redis会报''unable to require openssl...''的报错，因此需要继续像安装zlib一样安装openssl。

网址https://www.openssl.org/source/，解压后进入安装目录，执行以下命令，编译安装执行时间比较长。

[root@node01 /home/software/openssl-1.0.2t]# ./config -fPIC --prefix=/usr/local/openssl enable-shared
[root@node01 /home/software/openssl-1.0.2t]# ./config -t
# 这里会编译很久
[root@node01 /home/software/openssl-1.0.2t]# make && make install

（14）还需进入ruby根目录/ext/openssl下，执行ruby extconf.rb命令，也会在当前目录下生成Makefile文件，执行后光荣的报错了。

 
[root@node01 /home/software/ruby-2.3.1/ext/openssl]# ruby extconf.rb  --with-openssl-include=/usr/local/openssl/include/ --with-openssl-lib=/usr/local/openssl/lib
[root@node01 /home/software/ruby-2.3.1/ext/openssl]# make && make install

（15）类似上面安装zlib，也将Makefile文件的\$(top_srcdir)全部改成"../.."，这里在使用了":%s/\${top_srcdir}/..\/../g"进行了全局替换。再次编译安装就通过了。

（16）到这里，就差不多可以了，切换目录到redis根目录，然后执行gem install redis，再次报错。没辙了，网上找了一种土办法，先让安装成功。

[root@node01 /home/software/redis-3.2.11]# gem install redis
ERROR:  Could not find a valid gem 'redis' (>= 0), here is why:
          Unable to download data from https://rubygems.org/ - SSL_connect returned=1 errno=0 state=error: certificate verify failed (https://api.rubygems.org/specs.4.8.gz)

临时办法，大概意思让https连接改成http连接。

[root@node01 /home/software/redis-3.2.11]# gem sources -r https://rubygems.org
source https://rubygems.org not present in cache
[root@node01 /home/software/redis-3.2.11]# gem sources -a http://rubygems.org
https://rubygems.org is recommended for security over http://rubygems.org

Do you want to add this insecure source? [yn]  y
http://rubygems.org added to sources
You have new mail in /var/spool/mail/root

继续执行命令，终于ok了，可谓一波三折，这个方法只能作为参考，换一个redis或ruby又不知道会发生什么报错。

# 继续
[root@node01 /home/software/redis-3.2.11/src]# gem install redis
Successfully installed redis-3.2.1
Parsing documentation for redis-3.2.1
Installing ri documentation for redis-3.2.1
Done installing documentation for redis after 0 seconds #成功了
WARNING:  Unable to pull data from 'https://rubygems.org/': SSL_connect returned=1 errno=0 state=error: certificate verify failed (https://api.rubygems.org/specs.4.8.gz)
1 gem installed
You have new mail in /var/spool/mail/root
# 试运行redis-trib.rb脚本，终于ok了
[root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb
Usage: redis-trib <command> <options> <arguments ...>

  create          host1:port1 ... hostN:portN
                  --replicas <arg>
  check           host:port
  info            host:port
  fix             host:port
                  --timeout <arg>
  reshard         host:port
                  --from <arg>
                  --to <arg>
                  --slots <arg>
                  --yes
                  --timeout <arg>
                  --pipeline <arg>
  rebalance       host:port
                  --weight <arg>
                  --auto-weights
                  --use-empty-masters
                  --timeout <arg>
...省略

（17）前面准备好了后，就可以使用ruby脚本来开始构建集群了。上面启动后的redis节点都是各自为政互不联系，接下来先将8000、8001、8002作为主节点连起来，并分配槽道号，本次使用默认分配。

# 使用create
[root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb create 192.168.200.140:8000 192.168.200.140:8001 192.168.200.140:8002
>>> Creating cluster
>>> Performing hash slots allocation on 3 nodes...
Using 3 masters:
192.168.200.140:8000
192.168.200.140:8001
192.168.200.140:8002
M: dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000
   slots:0-5460 (5461 slots) master
M: aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001
   slots:5461-10922 (5462 slots) master
M: 719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002
   slots:10923-16383 (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
>>> Performing Cluster Check (using node 192.168.200.140:8000)
M: dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000
   slots:0-5460 (5461 slots) master
   0 additional replica(s)
M: aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001
   slots:5461-10922 (5462 slots) master
   0 additional replica(s)
M: 719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002
   slots:10923-16383 (5461 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
You have new mail in /var/spool/mail/root
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000
# 查看发现集群状态已变成ok，说明槽道没有一个没分配
127.0.0.1:8000> cluster into
(error) ERR Wrong CLUSTER subcommand or number of arguments
127.0.0.1:8000> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:3
cluster_size:3
cluster_current_epoch:3
cluster_my_epoch:1
cluster_stats_messages_sent:68
cluster_stats_messages_received:68
# 三个主节点都分配了槽道
127.0.0.1:8000> cluster nodes
# 节点信息
# 第一个为节点id，用40位的十六进制表示
# 第二个为节点ip+端口
# 第三个为角色
# 第四个为主节点id，没有主节点就用'-'表示
# 第五个是与操作相关的时间戳
# 第六个代表序号
# 第七个代表连接状态
# 第八个代表槽道号区间
aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001 master - 0 1576145625541 2 connected 5461-10922
719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002 master - 0 1576145623522 3 connected 10923-16383
dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000 myself,master - 0 0 1 connected 0-5460

（18）将8003挂接到8000，8004挂接到8001，8005挂接到8002成为从节点，使用add node命令，这里需要用到两个选项，一个是--salve，代表新加的节点是从，一个是--master-id，指定需要添加的主是谁，最后依次添加slaveHost:slavePort existHost:existPort，后面的节点可以是集群中任意一个集群的已知节点。

# 以添加8003为例 ，注意主节点可以是任意一个主节点，不一定是当前从的主节点
[root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb add-node --slave --master-id dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8003 192.168.200.140:8000
>>> Adding node 192.168.200.140:8003 to cluster 192.168.200.140:8000
>>> Performing Cluster Check (using node 192.168.200.140:8000)
M: dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000
   slots:0-5460 (5461 slots) master
   0 additional replica(s)
M: aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001
   slots:5461-10922 (5462 slots) master
   0 additional replica(s)
M: 719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002
   slots:10923-16383 (5461 slots) master
   0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
# 底层还是调用redis原生命令cluster meet
>>> Send CLUSTER MEET to node 192.168.200.140:8003 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 192.168.200.140:8000.
[OK] New node added correctly.
You have new mail in /var/spool/mail/root

登录集群客户端查看集群信息，发现搭建成功，实现了三主三从redis集群的搭建。

127.0.0.1:8000> cluster nodes
aa79ace502e5369236b62ed61c0eb43733ddcbde 192.168.200.140:8001 master - 0 1576152242486 2 connected 5461-10922
dc0e2e16888426089de31466ff398679eec81b86 192.168.200.140:8000 myself,master - 0 0 1 connected 0-5460
824a999176c12eb6ebd944b75944fd97c65b579d 192.168.200.140:8005 slave 719d1dd412faf6b4e1eb348c65c329a262e393e1 0 1576152244002 3 connected
312accb9be27c48c107a19ab501a75a8c5d321c6 192.168.200.140:8004 slave aa79ace502e5369236b62ed61c0eb43733ddcbde 0 1576152239461 2 connected
ada1903bcb5744900916151dd385936ceb6ce049 192.168.200.140:8003 slave dc0e2e16888426089de31466ff398679eec81b86 0 1576152243496 1 connected
719d1dd412faf6b4e1eb348c65c329a262e393e1 192.168.200.140:8002 master - 0 1576152244507 3 connected 10923-16383

ruby脚本简化搭建

可以看出，这个搭建比较繁琐，如果解决了ruby的报错问题，只需要一个命令就可以完成三主三从redis集群的搭建，下面记录一下。

（1）停止redis所有进程。

[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 shutdown
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8001 shutdown
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8002 shutdown
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8003 shutdown
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8004 shutdown
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8005 shutdown
[root@node01 /home/software/redis-3.2.11]# ps -ef|grep redis
root      3657 18054  0 20:13 pts/2    00:00:00 grep redis

（2）删除rdb和aof持久化文件，删除集群节点状态记录文件。

[root@node01 /home/software/redis-3.2.11]# rm -rf appendonly800*
[root@node01 /home/software/redis-3.2.11]# rm -rf dump800*
[root@node01 /home/software/redis-3.2.11]# rm -rf nodes-800*

（3）再次启动6个redis节点。

# 启动
[root@node01 /home/software/redis-3.2.11]# redis-server 8000/redis.conf
[root@node01 /home/software/redis-3.2.11]# redis-server 8001/redis.conf
[root@node01 /home/software/redis-3.2.11]# redis-server 8002/redis.conf
[root@node01 /home/software/redis-3.2.11]# redis-server 8003/redis.conf
[root@node01 /home/software/redis-3.2.11]# redis-server 8004/redis.conf
[root@node01 /home/software/redis-3.2.11]# redis-server 8005/redis.conf
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000
# 还没分配槽道，redis集群状态fail
127.0.0.1:8000> cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0

（4）使用ruby脚本src/redis-trib.rb create --replicas 1 六个的节点ip:端口，这一条命令就可以自动搭建了三主三从。

# 一个命令
[root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb create --replicas 1 192.168.200.140:8000 192.168.200.140:8001 192.168.200.140:8002 192.168.200.140:8003 192.168.200.140:8004 192.168.200.140:8005
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.200.140:8000
192.168.200.140:8001
192.168.200.140:8002
Adding replica 192.168.200.140:8003 to 192.168.200.140:8000
Adding replica 192.168.200.140:8004 to 192.168.200.140:8001
Adding replica 192.168.200.140:8005 to 192.168.200.140:8002
M: 6537ac1cb42f209600ea20b1d6310e7b789257d4 192.168.200.140:8000
   slots:0-5460 (5461 slots) master
M: 5da79737cf493085724a689bdb5bd878c9121c07 192.168.200.140:8001
   slots:5461-10922 (5462 slots) master
M: 49f40898d2d18cc162b00e5f69593f0b91af5766 192.168.200.140:8002
   slots:10923-16383 (5461 slots) master
S: a92a3eb2cfff488e0af6ca07d9b612f2f3595c0a 192.168.200.140:8003
   replicates 6537ac1cb42f209600ea20b1d6310e7b789257d4
S: ea41d1e7a41e5c5b19baa8e6cbac15b7d4d2bdb8 192.168.200.140:8004
   replicates 5da79737cf493085724a689bdb5bd878c9121c07
S: bede8e5c11fe947cbaf0e435d822057a7b0fe414 192.168.200.140:8005
   replicates 49f40898d2d18cc162b00e5f69593f0b91af5766
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 192.168.200.140:8000)
M: 6537ac1cb42f209600ea20b1d6310e7b789257d4 192.168.200.140:8000
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
M: 49f40898d2d18cc162b00e5f69593f0b91af5766 192.168.200.140:8002
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: bede8e5c11fe947cbaf0e435d822057a7b0fe414 192.168.200.140:8005
   slots: (0 slots) slave
   replicates 49f40898d2d18cc162b00e5f69593f0b91af5766
M: 5da79737cf493085724a689bdb5bd878c9121c07 192.168.200.140:8001
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: a92a3eb2cfff488e0af6ca07d9b612f2f3595c0a 192.168.200.140:8003
   slots: (0 slots) slave
   replicates 6537ac1cb42f209600ea20b1d6310e7b789257d4
S: ea41d1e7a41e5c5b19baa8e6cbac15b7d4d2bdb8 192.168.200.140:8004
   slots: (0 slots) slave
   replicates 5da79737cf493085724a689bdb5bd878c9121c07
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
You have new mail in /var/spool/mail/root
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000
# 结果一样
127.0.0.1:8000> cluster nodes
49f40898d2d18cc162b00e5f69593f0b91af5766 192.168.200.140:8002 master - 0 1576153283083 3 connected 10923-16383
bede8e5c11fe947cbaf0e435d822057a7b0fe414 192.168.200.140:8005 slave 49f40898d2d18cc162b00e5f69593f0b91af5766 0 1576153285101 6 connected
5da79737cf493085724a689bdb5bd878c9121c07 192.168.200.140:8001 master - 0 1576153286109 2 connected 5461-10922
6537ac1cb42f209600ea20b1d6310e7b789257d4 192.168.200.140:8000 myself,master - 0 0 1 connected 0-5460
a92a3eb2cfff488e0af6ca07d9b612f2f3595c0a 192.168.200.140:8003 slave 6537ac1cb42f209600ea20b1d6310e7b789257d4 0 1576153283585 4 connected
ea41d1e7a41e5c5b19baa8e6cbac15b7d4d2bdb8 192.168.200.140:8004 slave 5da79737cf493085724a689bdb5bd878c9121c07 0 1576153284091 5 connected

redis cluster原生命令搭建

以上两种方法，都是依赖ruby脚本，其实它还是基于redis cluster的命令来搭建完成，使用这种方式搭建能更好的理解搭建中经历了什么。

（1）提前准备6个启动的节点，前提也需删除持久化文件，以及节点状态记录文件，跟上面一样。

（2）登录任何一个节点，和其他5个进行meet，这里使用8000的客户端来完成，meet完后任意一个节点都认为自己是主节点。

# 使用cluster meet命令
192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8001
OK
192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8002
OK
192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8003
OK
192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8004
OK
192.168.200.140:8000> CLUSTER MEET 192.168.200.140 8005
OK
# 登录任何一个节点查看，都能得到如下信息
192.168.200.140:8000> cluster nodes
fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 master - 0 1576155518818 0 connected
ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 myself,master - 0 0 4 connected
2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 master - 0 1576155517810 5 connected
231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576155514284 1 connected
7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576155515791 2 connected
2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 master - 0 1576155516800 3 connected

（3）给8000，8001和8002分配槽道号，让其作为主节点。分配槽道号的命令为cluster addslots 槽道号...。分配完成后不管登录哪个节点都能看到槽道号信息，

# 槽道号之间有多个需要用空格隔开
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 -h 192.168.200.140
192.168.200.140:8000> CLUSTER ADDSLOTS 0 1 2 3 4 5
OK
# 可以看到8000的节点分配了0-5这6个槽道号
192.168.200.140:8000> cluster nodes
fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 master - 0 1576155898158 0 connected
ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 myself,master - 0 0 4 connected 0-5
2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 master - 0 1576155896137 5 connected
231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576155894119 1 connected
7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576155897147 2 connected
2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 master - 0 1576155895128 3 connected

一个个写来分配会比较费力气，可以写一个shell脚本来批量分配槽道号，使用for循环来实现。

shell脚本

#分配槽道号的脚本
#!/bin/bash
# 8000节点
for slot in {6..5460}
do redis-cli -c -p 8000 -h 192.168.200.140 cluster addslots $slot
done
# 8001节点
for slot in {5461..10922}
do redis-cli -c -p 8001 -h 192.168.200.140 cluster addslots $slot
done
# 8002节点
for slot in {10923..16383}
do redis-cli -c -p 8002 -h 192.168.200.140 cluster addslots $slot
done
# 执行完打印出来结果
echo "分配槽道号结束"

运行脚本执行完结果，ok。

登录一个节点查看集群槽道信息，发现跟分配的预想一样。

[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8001 -h 192.168.200.140
192.168.200.140:8001> cluster nodes
# 8001结果，与上面脚本一样结果
231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 myself,master - 0 0 1 connected 5461-10922 
fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 master - 0 1576157229386 0 connected
ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 master - 0 1576157228376 4 connected 0-5460
7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576157229386 2 connected 10923-16383
2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 master - 0 1576157227368 3 connected
2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 master - 0 1576157230393 5 connected

（4）将8003、8004和8005角色转换为从节点，使用cluster replicate 主节点id 命令。最后查看集群信息，发现成功的分配了槽道号。

[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8003 -h 192.168.200.140
# 成为8000的从节点
192.168.200.140:8003> cluster replicate ced952bc6a47c38756bbc48d85a5f601db620b79
OK
192.168.200.140:8003> quit
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8004 -h 192.168.200.140
# 成为8001的从节点
192.168.200.140:8004> cluster replicate 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83
OK
192.168.200.140:8004> quit
You have new mail in /var/spool/mail/root
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8005 -h 192.168.200.140
# 成为8002的从节点
192.168.200.140:8005> cluster replicate 7ce388bde879f686fc3c8491175397ca20405565
OK
# 结果ok
192.168.200.140:8005> cluster nodes
2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 myself,slave 7ce388bde879f686fc3c8491175397ca20405565 0 0 5 connected
7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576157733223 2 connected 10923-16383
2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 slave ced952bc6a47c38756bbc48d85a5f601db620b79 0 1576157732216 4 connected
ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 master - 0 1576157735238 4 connected 0-5460
231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576157736242 1 connected 5461-10922
fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slave 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 0 1576157734230 1 connected

最后集群状态为ok，槽道全部分配完成，形成了三主三从的结构。

192.168.200.140:8005> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
# 6个节点
cluster_known_nodes:6
# 3个主
cluster_size:3
cluster_current_epoch:5
cluster_my_epoch:2
cluster_stats_messages_sent:5093
cluster_stats_messages_received:5093

redis cluster高可用

现在在8000节点上保存数据，然后将8000的主节点宕机，8003从节点会顶替上来成为主节点，并获取到8000的槽道信息。

[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8000 -h 192.168.200.140
192.168.200.140:8000> set age 28
OK
192.168.200.140:8000> get age
"28"
192.168.200.140:8000> quit
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8003 -h 192.168.200.140
192.168.200.140:8003> get age
-> Redirected to slot [741] located at 192.168.200.140:8000
"28"
192.168.200.140:8000> shutdown
not connected> quit
You have new mail in /var/spool/mail/root
[root@node01 /home/software/redis-3.2.11]# redis-cli -c -p 8003 -h 192.168.200.140
# 8000宕机后，8003暂时还是从
192.168.200.140:8003> cluster nodes
fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slave 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 0 1576158134892 1 connected
2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 myself,slave ced952bc6a47c38756bbc48d85a5f601db620b79 0 0 3 connected
ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 master - 1576158120865 1576158119755 4 disconnected 0-5460
2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 slave 7ce388bde879f686fc3c8491175397ca20405565 0 1576158132873 5 connected
231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576158133883 1 connected 5461-10922
7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576158131864 2 connected 10923-16383
# 过一会，8003变为主，并且获得槽道信息
192.168.200.140:8003> cluster nodes
fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slave 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 0 1576158149019 1 connected
2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 myself,master - 0 0 6 connected 0-5460
ced952bc6a47c38756bbc48d85a5f601db620b79 192.168.200.140:8000 master,fail - 1576158120865 1576158119755 4 disconnected
2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 slave 7ce388bde879f686fc3c8491175397ca20405565 0 1576158148008 5 connected
231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576158146999 1 connected 5461-10922
7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576158150028 2 connected 10923-16383
# 数据也备份了
192.168.200.140:8003> get age
"28"

redis cluster添加和删除节点

再补充添加和删除节点的操作，关于删除节点，它只能删除从节点或没有槽道管理权的节点，由于8000已经没有了槽道管理权，将8000重启后，下面使用ruby脚本删除，命令为del-node ip:port 节点id。

# 确认8000开启
[root@node01 /home/software/redis-3.2.11]# ps -ef|grep redis
root      4170     1  0 20:51 ?        00:00:13 redis-server *:8001 [cluster]
root      4174     1  0 20:51 ?        00:00:11 redis-server *:8002 [cluster]
root      4178     1  0 20:51 ?        00:00:08 redis-server *:8003 [cluster]
root      4182     1  0 20:51 ?        00:00:08 redis-server *:8004 [cluster]
root      4186     1  0 20:51 ?        00:00:08 redis-server *:8005 [cluster]
root      7231     1  0 22:09 ?        00:00:00 redis-server *:8000 [cluster]
root      7298 18054  0 22:13 pts/2    00:00:00 grep redis
# 删除8000
[root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb del-node 192.168.200.140:8000 94f634b2d364d560d3f8fb6e88fa874cf2b493f6
>>> Removing node 94f634b2d364d560d3f8fb6e88fa874cf2b493f6 from cluster 192.168.200.140:8000
>>> Sending CLUSTER FORGET messages to the cluster...
# 关闭节点
>>> SHUTDOWN the node.
You have new mail in /var/spool/mail/root

发现删除节点后，还自动将节点关闭了，因为8000节点的状态记录文件nodes-8000.conf还记录着8000是连接状态，但是其他节点的这个文件已经没了8000的信息，为了集群数据的不混乱，需要将8000立即关闭。

既然删除了，尝试启动8000后添加到集群，需要使用add-node newHost:newPort existHost:existPort命令，类似添加从节点命令只是没有选项。发现启动8000后再添加进集群会报错。

# 确认开启8000
[root@node01 /home/software/redis-3.2.11]# ps -ef|grep redis
root      4170     1  0 20:51 ?        00:00:14 redis-server *:8001 [cluster]
root      4174     1  0 20:51 ?        00:00:11 redis-server *:8002 [cluster]
root      4178     1  0 20:51 ?        00:00:08 redis-server *:8003 [cluster]
root      4182     1  0 20:51 ?        00:00:08 redis-server *:8004 [cluster]
root      4186     1  0 20:51 ?        00:00:08 redis-server *:8005 [cluster]
root      7400     1  0 22:21 ?        00:00:00 redis-server *:8000 [cluster]
root      7422 18054  0 22:22 pts/2    00:00:00 grep redis
You have new mail in /var/spool/mail/root
# 添加8000到集群
[root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb add-node 192.168.200.140:8000 192.168.200.140:8001
>>> Adding node 192.168.200.140:8000 to cluster 192.168.200.140:8001
>>> Performing Cluster Check (using node 192.168.200.140:8001)
M: 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004
   slots: (0 slots) slave
   replicates 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83
M: 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
M: 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003
   slots:0-5460 (5461 slots) master
   0 additional replica(s)
S: 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005
   slots: (0 slots) slave
   replicates 7ce388bde879f686fc3c8491175397ca20405565
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
# 提示报错，提示8000可能有数据，或者8000里有其他集群的信息
[ERR] Node 192.168.200.140:8000 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

这里删除了8000的rdb和aof持久化文件，以及状态文件后，重启8000节点再次尝试添加进去集群发现成功，但是添加进去的节点为没有槽道管理权的主节点。

[root@node01 /home/software/redis-3.2.11]# src/redis-trib.rb add-node 192.168.200.140:8000 192.168.200.140:8001
>>> Adding node 192.168.200.140:8000 to cluster 192.168.200.140:8001
>>> Performing Cluster Check (using node 192.168.200.140:8001)
M: 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004
   slots: (0 slots) slave
   replicates 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83
M: 7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
M: 2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003
   slots:0-5460 (5461 slots) master
   0 additional replica(s)
S: 2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005
   slots: (0 slots) slave
   replicates 7ce388bde879f686fc3c8491175397ca20405565
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.200.140:8000 to make it join the cluster.
[OK] New node added correctly.

8000的信息

fd4c160edc74536d79ea29d239dca43275ec6b5a 192.168.200.140:8004 slave 231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 0 1576161337438 1 connected
# 没有槽道管理权的主节点
c41dbe9595ae83725d1322b032736fd198b26c49 192.168.200.140:8000 myself,master - 0 0 0 connected
2c52d95c3d6d4c396469a81edfc1493d984e0f2d 192.168.200.140:8005 slave 7ce388bde879f686fc3c8491175397ca20405565 0 1576161336432 2 connected
231fe9df31dc1ccf7cca5ae2fb2313979cd6fa83 192.168.200.140:8001 master - 0 1576161336432 1 connected 5461-10922
2e0f23d703874db80373f28b1be8c13f9de4fe6b 192.168.200.140:8003 master - 0 1576161339456 6 connected 0-5460
7ce388bde879f686fc3c8491175397ca20405565 192.168.200.140:8002 master - 0 1576161338448 2 connected 10923-16383

以上是redis-cluster集群搭建的基本知识，后面继续补充槽道知识。

参考博文：

（1）《Redis设计与实践》

（2）https://blog.csdn.net/qq_26710443/article/details/82724268 ruby安装

（3）https://www.jianshu.com/p/c38369097448

（4）https://www.cnblogs.com/xuliangxing/p/7146868.html ruby脚本不能执行解决，报错主要参考文

（5）https://blog.csdn.net/ck3207/article/details/90404952 openssl无法编译

posted @ 2019-12-12 23:23 斐波那切阅读(5349) 评论(0) 编辑收藏举报

刷新页面返回顶部