redis-cluster 集群增加节点

分布式存储机制-槽

【1】Redis Cluster 在设计中没有使用一致性哈希（Consistency Hashing），而是使用数据分片（Sharding）引入哈希槽
【2】Redis Cluster 把所有的节点映射到 [0-16383] slot 槽上，cluster 负责维护 node<->slot<->value 三者之间的关系。
【3】Redis 集群中内置了 16384 个哈希槽，当需要在 Redis 集群中放置一个 key-value 时，redis 先将 key 使用 CRC16 算法算出一个结果，然后把结果对 16384 求余数，这样每个 key 都会对应一个编号 0-16383 之间的哈希槽，redis 会根据节点数量大致均等的将哈希槽映射到不同的节点上。集群之间通过一种特殊的二进制协议交互集群信息。

例如，当有三个节点时，槽分布的值如下：
节点1： 0-5460
节点2： 5461-10921
节点3： 10922-16383

【4】Redis Cluster 允许用户强制把某个 key 挂在特定槽位上。通过在 key 字符串里面嵌入 tag 标记，这就可以强制 key 所挂的槽位等于 tag 所在的槽位。
【5】客户端为了可以直接定位某个具体的 key 所在的节点，需要缓存槽位相关信息，从而实现快速定位。同时因为客户端与服务端的槽位可能不一致，还需要纠正机制来实现槽位信息的校验调整。
【6】Redis Cluster 的每个节点会将集群的配置信息持久化到配置文件中，所以必须确保配置文件可写，而且尽量不要依靠人工修改配置文件。
【7】ClusterNode 数据结构中的 slots和 numslots属性记录了节点负责处理哪些槽。其中，slot属性是一个二进制位数组（bitarray），其长度为16384/8=2048 Byte，共包含16384个二进制位。集群中的 Master节点用bit（0和1）来标识是否拥有某个槽。比如，对于编号为1的槽，Master只要判断序列第二位（索引从0开始）的值是不是1即可，时间复杂度为O(1)。
———————————————

1.插槽（slot）

1.1 插槽的分配：

整个Redis提供了16384个插槽，也就是说集群中的每个节点分得的插槽数总和为16384。

./redis-trib.rb 脚本实现了是将16384个插槽平均分配给了N个节点。

1.2 数据如何保存到集群中

set abc 12345命令时，redis是如何将数据保存到集群中的执行步骤如下：

（1）接收命令set abc 12345

（2）通过key（abc）计算出插槽值，然后根据插槽值找到对应的节点。（abc的插槽值为：7638）

（3）重定向到拥有7638插槽的节点执行命令

插槽和key的关系：

通过key的有效部分使用CRC16算法计算出哈希值，再将哈希值对16384取余，得到插槽值。

2.向集群中新增节点

2.1新增一个master节点，确保该节点没有存储过数据。

cp -r /usr/local/redis-cluster/nodes-7000 nodes-7006 #复制一个节点
vi nodes-7006/nodes-7006.conf #修改配置文件

port 7006
#在不同的服务器和nodes-xx中，端口也不同
cluster-enabled yes bind 127.0.0.1
# daemonize yes #redis后台运行
cluster-config-file nodes-7006.conf
cluster-node-timeout 5000
appendonly yes

2.2启动7006

[root@localhost redis40]# src/redis-server /usr/local/redis-cluster/nodes-7006/nodes-7006.conf

2.3使用 redis-trib.rb，添加节点

语法 ./redis-trib.rb add-node new_host:new_port existing_host:existing_port

命令：

./src/redis-trib.rb add-node 127.0.0.1:7006 127.0.0.1:7000

注：add-node是加入集群节点，127.0.0.1:7006 为要加入的节点，127.0.0.1:7000 表示加入集群中的一个节点，用来辨识是哪个集群，理论上那个集群的节点都可以。

watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpeWlqaWFueGlhbg_size_16_color_FFFFFF_t_70

已经添加成功！查看集群信息：

watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpeWlqaWFueGlhbg_size_16_color_FFFFFF_t_70 1

2.4分配插槽数

使用 redis-trib.rb 命令从其他三个节点中转移1000个插槽到7006

（1）、连接集群中的任意节点

./src/redis-trib.rb reshard 127.0.0.1:7006

（2）、需要移动多少个插槽，输入1000

（3）、填写需要接收插槽的ID，输入 6385 的节点ID：6d9eb1d27cc7c1d3104b1d6328651584fa35af70

（4）、all：从所有的master节点平均取， done：从某一个master节点中取插槽，最后用done结束，输入all

（5）、确认分配插槽，输入yes

（6）、插槽分配完成，查看节点信息：其他master上的插槽已经都转移到7006上了

watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpeWlqaWFueGlhbg_size_16_color_FFFFFF_t_70 2

3.删除节点

./src/redis-trib.rb del-node 127.0.0.1:7006 6d9eb1d27cc7c1d3104b1d6328651584fa35af70

删除已经占有hash槽的结点会失败，需要将该结点占用的hash槽分配出去

4.重新分片

把某个节点的槽分配出去。

访问集群任意节点 ./src/redis-cli -c -p 7006 cluster nodes

watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpeWlqaWFueGlhbg_size_16_color_FFFFFF_t_70 3

4.1把这个节点拥有的slots全部迁移出去

watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpeWlqaWFueGlhbg_size_16_color_FFFFFF_t_70 4

How many slots do you want to move (from 1 to 16384)?16384 //输入一个大于或等于7001节点所拥有的slots数的数即可.

将7006节点上的插槽转移到7000上。

4.2删除j7006节点

4.3查看现有节点，集群中已不存7006节点

节点集群部署、节点的增加及删除和优化实战

1 Redis Cluster集群介绍

Redis主从，解决了Redis单点问题，但是没有实现Redis状态监控及故障自动切换，于是后来又引入了sentinel（哨兵）解决此问题。但是依然没能解决数据的一个并发读写的问题，那么Redis集群就是来解决此问题的，它是一个提供在多个Redis节点间共享数据的程序集。

Redis集群通过分区来提供一定程度的可用性，在实际环境中当某个节点宕机或者不可达的情况下可以继续处理业务。

1.1 Redis集群的优势

自动分割数据到不同的节点上；
整个集群的部分节点失败或者不可达的情况下能够继续处理命令。

要让集群正常运作至少需要三个主节点，为了实现主节点的高可用，强烈建议使用六个节点：其中三个为主节点，而其余三个则是各个主节点的从节点。

1.2 Redis集群的工作原理

Redis 集群没有使用一致性hash, 而是引入了哈希槽的概念。

Redis 集群有16384个哈希槽,每个key通过CRC16校验后对16384取模来决定放置哪个槽。集群的每个节点负责一部分hash槽,举个例子,比如当前集群有3个节点,那么:

节点 A 包含 0 到 5460号哈希槽.
节点 B 包含5461到10922号哈希槽.
节点 C 包含10923到16383号哈希槽.

这种结构很容易添加或者删除节点，比如如果我想新添加个节点D，我需要从节点 A, B, C中分配部分哈希槽到节点D上。如果我想移除节点A，需要将节点A中的哈希槽移到节点B和节点C上，然后将没有任何哈希槽的节点A从集群中移除即可。由于从一个节点将哈希槽移动到另一个节点并不会停止服务，所以无论添加、删除或者改变某个节点的哈希槽的数量都不会造成集群不可用的状态。

1.4 Redis集群的主从复制模型

为了使在部分节点失败或者大部分节点无法通信的情况下集群仍然可用，所以集群使用了主从复制模型,每个节点都会有N-1个复制品。

在上面具有A，B，C三个节点的集群，在没有复制模型的情况下，如果节点B失败了，那么整个集群就会以为缺少5461到10922这个范围的哈希槽而不可用。

然而如果在集群创建的时候（或者过一段时间）我们为每个节点添加一个从节点A1，B1，C1,那么整个集群便有三个master节点和三个slave节点组成，这样在节点B失败后，集群便会选举B1为新的主节点继续服务，整个集群便不会因为哈希槽找不到而不可用了，不过当B和B1都失败后，集群是不可用的。

2 Redis cluster部署

2.1 项目环境

节点	ip	端口	OS版本	redis版本
node01	192.168.1.181	7000	CentOS 7.6	6.2.1
node02	192.168.1.182	7001	CentOS 7.6	6.2.1
node03	192.168.1.183	7002	CentOS 7.6	6.2.1
node04	192.168.1.184	7003	CentOS 7.6	6.2.1
node05	192.168.1.185	7004	CentOS 7.6	6.2.1
node06	192.168.1.186	7005	CentOS 7.6	6.2.1

2.2 Redis安装(6节点分别执行，以节点1为例)

# 下载解压
[root@node01 src]# cd usr/src
[root@node01 src]# wget http://download.Redis.io/edis-stable.tar.gz
[root@node01 src]# tar xf redis-stable.tar.gz
[root@node01 src]# cd redis-stable

# 编译安装Redis
[root@node01 redis-stable]# make PREFIX=/usr/local/redis install

# 到这一步，Redis已经安装好了

# 设置Redis参数：
[root@node01 redis-stable]# echo "vm.overcommit_memory = 1" >> etc/sysctl.conf
[root@node01 redis-stable]# echo "net.core.somaxconn = 512" >> etc/sysctl.conf
[root@node01 redis-stable]# echo never > sys/kernel/mm/transparent_hugepage/enabled
[root@node01 redis-stable]# sysctl -p

# 设置radis环境变量：
[root@node01 ~]# cat > etc/profile.d/redis.sh << EOF
PATH=/usr/local/redis/bin:$PATH
export PATH
EOF
[root@node01 ~]#
[root@node01 ~]# source etc/profile
[root@node01 ~]#

2.3 创建Redis实例

2.3.1 创建节点1实例

用初始化脚本，快速创建实例配置文件，日志目录：

[root@node01 redis-stable]# usr/src/redis-stable/utils/install_server.sh
Welcome to the redis service installer
This script will help you easily set up a running Redis server

Please select the redis port for this instance: [6379] 7000
Please select the redis config file name [/etc/redis/7000.conf] usr/local/redis/7000.conf
Please select the redis log file name [/var/log/redis_7000.log] usr/local/redis/7000.log
Please select the data directory for this instance [/var/lib/redis/7000] usr/local/redis/7000
Please select the redis executable path [] usr/local/redis/bin/redis-server
Selected config:
Port           : 7000
Config file   : usr/local/redis/7000.conf
Log file       : usr/local/redis/7000.log
Data dir       : usr/local/redis/7000
Executable     : usr/local/redis/bin/redis-server
Cli Executable : usr/local/redis/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied tmp/7000.conf => etc/init.d/redis_7000
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!
[root@node01 redis-stable]#

2.3.2 创建节点2实例

用初始化脚本，快速创建实例配置文件，日志目录：

[root@node01 redis-stable]# usr/src/redis-stable/utils/install_server.sh
Welcome to the redis service installer
This script will help you easily set up a running Redis server

Please select the redis port for this instance: [6379] 7001
Please select the redis config file name [/etc/redis/7001.conf] usr/local/redis/7001.conf
Please select the redis log file name [/var/log/redis_7001.log] usr/local/redis/7001.log
Please select the data directory for this instance [/var/lib/redis/7001] usr/local/redis/7001
Please select the redis executable path [] usr/local/redis/bin/redis-server
Selected config:
Port           : 7001
Config file   : usr/local/redis/7001.conf
Log file       : usr/local/redis/7001.log
Data dir       : usr/local/redis/7001
Executable     : usr/local/redis/bin/redis-server
Cli Executable : usr/local/redis/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied tmp/7001.conf => etc/init.d/redis_7001
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!
[root@node01 redis-stable]#

2.3.3 创建节点3实例

用初始化脚本，快速创建实例配置文件，日志目录：

[root@node02 redis-stable]# usr/src/redis-stable/utils/install_server.sh
Welcome to the redis service installer
This script will help you easily set up a running Redis server

Please select the redis port for this instance: [6379] 7002
Please select the redis config file name [/etc/redis/7002.conf] usr/local/redis/7002.conf
Please select the redis log file name [/var/log/redis_7002.log] usr/local/redis/7002.log
Please select the data directory for this instance [/var/lib/redis/7002] ^C
[root@node02 redis-stable]# usr/src/redis-stable/utils/install_server.sh
Welcome to the redis service installer
This script will help you easily set up a running Redis server

Please select the redis port for this instance: [6379] 7002
Please select the redis config file name [/etc/redis/7002.conf] usr/local/redis/7002.conf
Please select the redis log file name [/var/log/redis_7002.log] usr/local/redis/7002.log
Please select the data directory for this instance [/var/lib/redis/7002] usr/local/redis/7002
Please select the redis executable path [] usr/local/redis/bin/redis-server
Selected config:
Port           : 7002
Config file   : usr/local/redis/7002.conf
Log file       : usr/local/redis/7002.log
Data dir       : usr/local/redis/7002
Executable     : usr/local/redis/bin/redis-server
Cli Executable : usr/local/redis/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied tmp/7002.conf => etc/init.d/redis_7002
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!
[root@node02 redis-stable]#

2.3.4 创建节点4实例

用初始化脚本，快速创建实例配置文件，日志目录：

[root@node02 redis-stable]# usr/src/redis-stable/utils/install_server.sh
Welcome to the redis service installer
This script will help you easily set up a running Redis server

Please select the redis port for this instance: [6379] 7003
Please select the redis config file name [/etc/redis/7003.conf] usr/local/redis/7003.conf
Please select the redis log file name [/var/log/redis_7003.log] usr/local/redis/7003.log
Please select the data directory for this instance [/var/lib/redis/7003] usr/local/redis/7003
Please select the redis executable path [] usr/local/redis/bin/redis-server
Selected config:
Port           : 7003
Config file   : usr/local/redis/7003.conf
Log file       : usr/local/redis/7003.log
Data dir       : usr/local/redis/7003
Executable     : usr/local/redis/bin/redis-server
Cli Executable : usr/local/redis/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied tmp/7003.conf => etc/init.d/redis_7003
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!
[root@node02 redis-stable]#

2.3.5 创建节点5实例

用初始化脚本，快速创建实例配置文件，日志目录：

[root@node03 redis-stable]# usr/src/redis-stable/utils/install_server.sh
Welcome to the redis service installer
This script will help you easily set up a running Redis server

Please select the redis port for this instance: [6379] 7004
Please select the redis config file name [/etc/redis/7004.conf] usr/local/redis/7004.conf
Please select the redis log file name [/var/log/redis_7004.log] usr/local/redis/7004.log
Please select the data directory for this instance [/var/lib/redis/7004] usr/local/redis/7004
Please select the redis executable path [] usr/local/redis/bin/redis-server
Selected config:
Port           : 7004
Config file   : usr/local/redis/7004.conf
Log file       : usr/local/redis/7004.log
Data dir       : usr/local/redis/7004
Executable     : usr/local/redis/bin/redis-server
Cli Executable : usr/local/redis/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied tmp/7004.conf => etc/init.d/redis_7004
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!
[root@node03 redis-stable]#

2.3.6 创建节点6实例

用初始化脚本，快速创建实例配置文件，日志目录：

[root@node03 redis-stable]# usr/src/redis-stable/utils/install_server.sh
Welcome to the redis service installer
This script will help you easily set up a running Redis server

Please select the redis port for this instance: [6379] 7005
Please select the redis config file name [/etc/redis/7005.conf] usr/local/redis/7005.conf
Please select the redis log file name [/var/log/redis_7005.log] usr/local/redis/7005.log
Please select the data directory for this instance [/var/lib/redis/7005] usr/local/redis/7005
Please select the redis executable path [] usr/local/redis/bin/redis-server
Selected config:
Port           : 7005
Config file   : usr/local/redis/7005.conf
Log file       : usr/local/redis/7005.log
Data dir       : usr/local/redis/7005
Executable     : usr/local/redis/bin/redis-server
Cli Executable : usr/local/redis/bin/redis-cli
Is this ok? Then press ENTER to go on or Ctrl-C to abort.
Copied tmp/7005.conf => etc/init.d/redis_7005
Installing service...
Successfully added to chkconfig!
Successfully added to runlevels 345!
Starting Redis server...
Installation successful!
[root@node03 redis-stable]#

2.4 开启Redis集群模式

在每个Redis实例的配置文件中，开启如下参数（以节点1为例）：

[root@node01 ~]# vi usr/local/redis/7000.conf
bind 0.0.0.0
cluster-enabled yes
cluster-config-file nodes-7000.conf
cluster-node-timeout 5000
appendonly yes
protected-mode no # 关闭保护模式
requirepass **** # master开启密码保护
masterauth **** # replica同master交互密码

2.5 重启Redis实例

[root@node01 ~]# etc/init.d/redis_7000 restart
Stopping ...
Redis stopped
Starting Redis server...
[root@node01 ~]#

[root@node01 ~]# etc/init.d/redis_7001 restart
Stopping ...
Redis stopped
Starting Redis server...
[root@node01 ~]#

[root@node02 ~]# etc/init.d/redis_7002 restart
Stopping ...
Redis stopped
Starting Redis server...
[root@node02 ~]#

[root@node02 ~]# etc/init.d/redis_7003 restart
Stopping ...
Redis stopped
Starting Redis server...
[root@node02 ~]#

[root@node03 ~]# etc/init.d/redis_7004 restart
Stopping ...
Redis stopped
Starting Redis server...
[root@node03 ~]#

[root@node03 ~]# etc/init.d/redis_7005 restart
Stopping ...
Redis stopped
Starting Redis server...
[root@node03 ~]#

2.6 创建集群

现在已经有了六个正在运行中的 Redis 实例，接下来就可以直接使用这些实例来创建集群，并为每个节点编写配置文件。

如果使用的是Redis 5以上的版本，可以直接使用redis-cli命令创建新集群，检查或重新硬化现有集群等等。

对于Redis版本3或4，使用Redis-trib.rb创建集群，它是一个Ruby程序。可以在Redis源代码的src目录下找到它。

# 创建集群：
[root@node01 ~]# usr/local/redis/bin/redis-cli --cluster create 192.168.1.181:7000 192.168.1.181:7001 \
192.168.1.182:7002 192.168.1.182:7003 192.168.1.183:7004 192.168.1.183:7005 \
--cluster-replicas 1

--cluster-replicas 1 表示为集群中的每个主节点创建一个从节点。

其中指定了6个Redis的ip：port,3个master3个slave。
以上命令回车后，当我们输入yes，redis-cli就会将这份配置应用到集群当中,让各个节点开始互相通讯,最后可以得到如下信息：
005 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.1.183:7004 to 192.168.1.181:7000
Adding replica 192.168.1.183:7005 to 192.168.1.181:7001
Adding replica 192.168.1.182:7003 to 192.168.1.182:7002
M: 8fffaca408475225eda34260a2793cdff0f7a5f2 192.168.1.181:7000
  slots:[0-5460] (5461 slots) master
M: 5277b27644e037f0d92e39a44dc6f40d7f422a3f 192.168.1.181:7001
  slots:[5461-10922] (5462 slots) master
M: d5ecca53d38c4746be746b4be8a7904e81da6529 192.168.1.182:7002
  slots:[10923-16383] (5461 slots) master
S: 4669f9ed7daa587c31de2dc2a0243be07049b8a6 192.168.1.182:7003
  replicates d5ecca53d38c4746be746b4be8a7904e81da6529
S: 7b088ee93f02b3cc0b9018ffee0eaf7035328503 192.168.1.183:7004
  replicates 8fffaca408475225eda34260a2793cdff0f7a5f2
S: aef97908c5b67a64f195f840e023e9187e19197a 192.168.1.183:7005
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
....
>>> Performing Cluster Check (using node 192.168.1.181:7000)
M: 8fffaca408475225eda34260a2793cdff0f7a5f2 192.168.1.181:7000
  slots:[0-5460] (5461 slots) master
  1 additional replica(s)
M: 5277b27644e037f0d92e39a44dc6f40d7f422a3f 192.168.1.181:7001
  slots:[5461-10922] (5462 slots) master
  1 additional replica(s)
S: 4669f9ed7daa587c31de2dc2a0243be07049b8a6 192.168.1.182:7003
  slots: (0 slots) slave
  replicates d5ecca53d38c4746be746b4be8a7904e81da6529
S: 7b088ee93f02b3cc0b9018ffee0eaf7035328503 192.168.1.183:7004
  slots: (0 slots) slave
  replicates 8fffaca408475225eda34260a2793cdff0f7a5f2
S: aef97908c5b67a64f195f840e023e9187e19197a 192.168.1.183:7005
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
M: d5ecca53d38c4746be746b4be8a7904e81da6529 192.168.1.182:7002
  slots:[10923-16383] (5461 slots) master
  1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@node01 ~]#

2.7 查看集群信息或状态

[root@node01 ~]# redis-cli -p 7000 cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:120
cluster_stats_messages_pong_sent:163
cluster_stats_messages_sent:283
cluster_stats_messages_ping_received:158
cluster_stats_messages_pong_received:120
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:283
[root@node01 ~]#

2.8 查看集群节点

[root@node01 ~]# redis-cli -p 7000 cluster nodes
5277b27644e037f0d92e39a44dc6f40d7f422a3f 192.168.1.181:7001@17001 master - 0 1593086224000 2 connected 5461-10922
4669f9ed7daa587c31de2dc2a0243be07049b8a6 192.168.1.182:7003@17003 slave d5ecca53d38c4746be746b4be8a7904e81da6529 0 1593086224398 4 connected
7b088ee93f02b3cc0b9018ffee0eaf7035328503 192.168.1.183:7004@17004 slave 8fffaca408475225eda34260a2793cdff0f7a5f2 0 1593086225911 5 connected
aef97908c5b67a64f195f840e023e9187e19197a 192.168.1.183:7005@17005 slave 5277b27644e037f0d92e39a44dc6f40d7f422a3f 0 1593086224000 6 connected
d5ecca53d38c4746be746b4be8a7904e81da6529 192.168.1.182:7002@17002 master - 0 1593086225000 3 connected 10923-16383
8fffaca408475225eda34260a2793cdff0f7a5f2 192.168.1.181:7000@17000 myself,master - 0 1593086225000 1 connected 0-5460
[root@node01 ~]#

2.9 删除节点

删除节点一定要注意，删除节点之前一定要先将数据移到其它节点，不能直接删除，不过也不用担心，即使有数据，不小心执行了删除节点指令，也会报有数据存在，不可以删除的错误。

# 查看状态：
[root@node01 ~]# redis-cli --cluster check 192.168.1.181:7000
192.168.1.181:7000 (8fffaca4...) -> 0 keys | 5461 slots | 1 slaves.
192.168.1.181:7001 (5277b276...) -> 0 keys | 5462 slots | 1 slaves.
192.168.1.182:7003 (4669f9ed...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.181:7000)
M: 8fffaca408475225eda34260a2793cdff0f7a5f2 192.168.1.181:7000
  slots:[0-5460] (5461 slots) master
  1 additional replica(s)
M: 5277b27644e037f0d92e39a44dc6f40d7f422a3f 192.168.1.181:7001
  slots:[5461-10922] (5462 slots) master
  1 additional replica(s)
M: 4669f9ed7daa587c31de2dc2a0243be07049b8a6 192.168.1.182:7003
  slots:[10923-16383] (5461 slots) master
  1 additional replica(s)
S: 7b088ee93f02b3cc0b9018ffee0eaf7035328503 192.168.1.183:7004
  slots: (0 slots) slave
  replicates 8fffaca408475225eda34260a2793cdff0f7a5f2
S: aef97908c5b67a64f195f840e023e9187e19197a 192.168.1.183:7005
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
S: d5ecca53d38c4746be746b4be8a7904e81da6529 192.168.1.182:7002
  slots: (0 slots) slave
  replicates 4669f9ed7daa587c31de2dc2a0243be07049b8a6
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@node01 ~]#

# 通过重新分片，将散列槽移到其它节点：
[root@node01 ~]# redis-cli --cluster reshard 192.168.1.181:7000
>>> Performing Cluster Check (using node 192.168.1.181:7000)
M: 8fffaca408475225eda34260a2793cdff0f7a5f2 192.168.1.181:7000
  slots:[0-5460] (5461 slots) master
  1 additional replica(s)
M: 5277b27644e037f0d92e39a44dc6f40d7f422a3f 192.168.1.181:7001
  slots:[5461-10922] (5462 slots) master
  1 additional replica(s)
M: 4669f9ed7daa587c31de2dc2a0243be07049b8a6 192.168.1.182:7003
  slots:[10923-16383] (5461 slots) master
  1 additional replica(s)
S: 7b088ee93f02b3cc0b9018ffee0eaf7035328503 192.168.1.183:7004
  slots: (0 slots) slave
  replicates 8fffaca408475225eda34260a2793cdff0f7a5f2
S: aef97908c5b67a64f195f840e023e9187e19197a 192.168.1.183:7005
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
S: d5ecca53d38c4746be746b4be8a7904e81da6529 192.168.1.182:7002
  slots: (0 slots) slave
  replicates 4669f9ed7daa587c31de2dc2a0243be07049b8a6
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 5461

# 用来接受散列槽的节点，可以任意指定：
What is the receiving node ID? 5277b27644e037f0d92e39a44dc6f40d7f422a3f
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
 
# 7000节点的id(意思是把该节点的散列槽分配给其它节点):
Source node #1: 8fffaca408475225eda34260a2793cdff0f7a5f2
Source node #2: done

# 查看状态：发现192.168.1.181:7000上的散列槽已经没有了。
[root@node01 ~]# redis-cli --cluster check 192.168.1.181:7000  
192.168.1.181:7000 (8fffaca4...) -> 0 keys | 0 slots | 0 slaves.
192.168.1.181:7001 (5277b276...) -> 0 keys | 10923 slots | 2 slaves.
192.168.1.182:7003 (4669f9ed...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.181:7000)
M: 8fffaca408475225eda34260a2793cdff0f7a5f2 192.168.1.181:7000
  slots: (0 slots) master
M: 5277b27644e037f0d92e39a44dc6f40d7f422a3f 192.168.1.181:7001
  slots:[0-10922] (10923 slots) master
  2 additional replica(s)
M: 4669f9ed7daa587c31de2dc2a0243be07049b8a6 192.168.1.182:7003
  slots:[10923-16383] (5461 slots) master
  1 additional replica(s)
S: 7b088ee93f02b3cc0b9018ffee0eaf7035328503 192.168.1.183:7004
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
S: aef97908c5b67a64f195f840e023e9187e19197a 192.168.1.183:7005
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
S: d5ecca53d38c4746be746b4be8a7904e81da6529 192.168.1.182:7002
  slots: (0 slots) slave
  replicates 4669f9ed7daa587c31de2dc2a0243be07049b8a6
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@node01 ~]#

# node01（192.168.1.181）已经没有了散列槽：
192.168.1.181:7000 (8fffaca4...) -> 0 keys | 0 slots | 0 slaves.
192.168.1.181:7001 (5277b276...) -> 0 keys | 10923 slots | 2 slaves.
192.168.1.182:7003 (4669f9ed...) -> 0 keys | 5461 slots | 1 slaves.

# 确定没问题了，执行下面命令，可以实现删除节点：
[root@node01 ~]# redis-cli --cluster del-node 192.168.1.181:7000 8fffaca408475225eda34260a2793cdff0f7a5f2
>>> Removing node 8fffaca408475225eda34260a2793cdff0f7a5f2 from cluster 192.168.1.181:7000
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@node01 ~]#

# 检查状态：发现192.168.1.181:7000节点已经没有了。
[root@node01 ~]# redis-cli --cluster check 192.168.1.181:7001  
192.168.1.181:7001 (5277b276...) -> 0 keys | 10923 slots | 2 slaves.
192.168.1.182:7003 (4669f9ed...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 2 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.181:7001)
M: 5277b27644e037f0d92e39a44dc6f40d7f422a3f 192.168.1.181:7001
  slots:[0-10922] (10923 slots) master
  2 additional replica(s)
S: 7b088ee93f02b3cc0b9018ffee0eaf7035328503 192.168.1.183:7004
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
S: d5ecca53d38c4746be746b4be8a7904e81da6529 192.168.1.182:7002
  slots: (0 slots) slave
  replicates 4669f9ed7daa587c31de2dc2a0243be07049b8a6
M: 4669f9ed7daa587c31de2dc2a0243be07049b8a6 192.168.1.182:7003
  slots:[10923-16383] (5461 slots) master
  1 additional replica(s)
S: aef97908c5b67a64f195f840e023e9187e19197a 192.168.1.183:7005
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@node01 ~]#

2.10 添加新的节点

添加节点前，一定要初始化好此节点：

# 添加节点
[root@node01 ~]# redis-cli --cluster add-node 192.168.1.181:7000 192.168.1.181:7001
>>> Adding node 192.168.1.181:7000 to cluster 192.168.1.181:7001
>>> Performing Cluster Check (using node 192.168.1.181:7001)
M: 5277b27644e037f0d92e39a44dc6f40d7f422a3f 192.168.1.181:7001
  slots:[0-10922] (10923 slots) master
  2 additional replica(s)
S: 7b088ee93f02b3cc0b9018ffee0eaf7035328503 192.168.1.183:7004
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
S: d5ecca53d38c4746be746b4be8a7904e81da6529 192.168.1.182:7002
  slots: (0 slots) slave
  replicates 4669f9ed7daa587c31de2dc2a0243be07049b8a6
M: 4669f9ed7daa587c31de2dc2a0243be07049b8a6 192.168.1.182:7003
  slots:[10923-16383] (5461 slots) master
  1 additional replica(s)
S: aef97908c5b67a64f195f840e023e9187e19197a 192.168.1.183:7005
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.1.181:7000 to make it join the cluster.
[OK] New node added correctly.
[root@node01 ~]#

# 看到以下信息，表示节点添加成功：
...
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.1.181:7000 to make it join the cluster.
[OK] New node added correctly.
[root@node01 ~]#

2.11 重新分片

虽然节点添加成功，但是还没有分配散列槽，需要重新分片，就是将其它节点上的部分散列槽移动到该节点上。

# 可以看到虽然7000新节点添加成功，但是没有分配散列槽：
[root@node01 ~]# redis-cli --cluster check 192.168.1.181:7000
192.168.1.181:7000 (b7a5add9...) -> 0 keys | 0 slots | 0 slaves.
192.168.1.181:7001 (5277b276...) -> 0 keys | 10923 slots | 2 slaves.
192.168.1.182:7003 (4669f9ed...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.181:7000)
M: b7a5add92ee8103a2032688c3a7728bc8db02b23 192.168.1.181:7000
  slots: (0 slots) master
M: 5277b27644e037f0d92e39a44dc6f40d7f422a3f 192.168.1.181:7001
  slots:[0-10922] (10923 slots) master
  2 additional replica(s)
S: 7b088ee93f02b3cc0b9018ffee0eaf7035328503 192.168.1.183:7004
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
S: aef97908c5b67a64f195f840e023e9187e19197a 192.168.1.183:7005
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
M: 4669f9ed7daa587c31de2dc2a0243be07049b8a6 192.168.1.182:7003
  slots:[10923-16383] (5461 slots) master
  1 additional replica(s)
S: d5ecca53d38c4746be746b4be8a7904e81da6529 192.168.1.182:7002
  slots: (0 slots) slave
  replicates 4669f9ed7daa587c31de2dc2a0243be07049b8a6
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@node01 ~]#

# 重新分配，只需指定一个节点，redis-cli将自动找到其它节点：
[root@node01 ~]# redis-cli --cluster reshard 192.168.1.181:7000
>>> Performing Cluster Check (using node 192.168.1.181:7000)
M: b7a5add92ee8103a2032688c3a7728bc8db02b23 192.168.1.181:7000
  slots: (0 slots) master
M: 5277b27644e037f0d92e39a44dc6f40d7f422a3f 192.168.1.181:7001
  slots:[0-10922] (10923 slots) master
  2 additional replica(s)
S: 7b088ee93f02b3cc0b9018ffee0eaf7035328503 192.168.1.183:7004
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
S: aef97908c5b67a64f195f840e023e9187e19197a 192.168.1.183:7005
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
M: 4669f9ed7daa587c31de2dc2a0243be07049b8a6 192.168.1.182:7003
  slots:[10923-16383] (5461 slots) master
  1 additional replica(s)
S: d5ecca53d38c4746be746b4be8a7904e81da6529 192.168.1.182:7002
  slots: (0 slots) slave
  replicates 4669f9ed7daa587c31de2dc2a0243be07049b8a6
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

首先指定分配多少散列槽

# 重新分配多少个散列槽，这里尝试重新设置5461个散列槽：
How many slots do you want to move (from 1 to 16384)? 5461

指定接收哈希槽的节点ID

# 指定接受的Redis节点ID，注意不是指定ip地址：
How many slots do you want to move (from 1 to 16384)? 5461
What is the receiving node ID? b7a5add92ee8103a2032688c3a7728bc8db02b23

指定从哪些节点获取散列槽

Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1: 5277b27644e037f0d92e39a44dc6f40d7f422a3f
Source node #2: done

# all：表示自动在所有节点进行分配。
# done：表示指定节点结束，一般用在移除节点时，指定完Source node #1: 节点ID，再输入done，表示输入结束。

是否确定分配

  Moving slot 5457 from 5277b27644e037f0d92e39a44dc6f40d7f422a3f
  Moving slot 5458 from 5277b27644e037f0d92e39a44dc6f40d7f422a3f
  Moving slot 5459 from 5277b27644e037f0d92e39a44dc6f40d7f422a3f
  Moving slot 5460 from 5277b27644e037f0d92e39a44dc6f40d7f422a3f
Do you want to proceed with the proposed reshard plan (yes/no)? yes

在最终确认之后，redis-cli开始执行重新分配

# 下面这个片段可以看到Redis正在从7001上移动散列槽到7000上：
...
Moving slot 1584 from 192.168.1.181:7001 to 192.168.1.181:7000:
Moving slot 1585 from 192.168.1.181:7001 to 192.168.1.181:7000:
Moving slot 1586 from 192.168.1.181:7001 to 192.168.1.181:7000:
Moving slot 1587 from 192.168.1.181:7001 to 192.168.1.181:7000:
Moving slot 1588 from 192.168.1.181:7001 to 192.168.1.181:7000:
Moving slot 1589 from 192.168.1.181:7001 to 192.168.1.181:7000:
Moving slot 1590 from 192.168.1.181:7001 to 192.168.1.181:7000:
Moving slot 1591 from 192.168.1.181:7001 to 192.168.1.181:7000:
Moving slot 1592 from 192.168.1.181:7001 to 192.168.1.181:7000:
Moving slot 1593 from 192.168.1.181:7001 to 192.168.1.181:7000:

重新分片正在进行中时，程序不受影响地运行。检查集群的运行状态：

# 可以看到散列槽已经分配了
[root@node01 ~]# redis-cli --cluster check 192.168.1.181:7000  
192.168.1.181:7000 (b7a5add9...) -> 0 keys | 5461 slots | 1 slaves.
192.168.1.181:7001 (5277b276...) -> 0 keys | 5462 slots | 1 slaves.
192.168.1.182:7003 (4669f9ed...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.181:7000)
M: b7a5add92ee8103a2032688c3a7728bc8db02b23 192.168.1.181:7000
  slots:[0-5460] (5461 slots) master
  1 additional replica(s)
M: 5277b27644e037f0d92e39a44dc6f40d7f422a3f 192.168.1.181:7001
  slots:[5461-10922] (5462 slots) master
  1 additional replica(s)
S: 7b088ee93f02b3cc0b9018ffee0eaf7035328503 192.168.1.183:7004
  slots: (0 slots) slave
  replicates b7a5add92ee8103a2032688c3a7728bc8db02b23
S: aef97908c5b67a64f195f840e023e9187e19197a 192.168.1.183:7005
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
M: 4669f9ed7daa587c31de2dc2a0243be07049b8a6 192.168.1.182:7003
  slots:[10923-16383] (5461 slots) master
  1 additional replica(s)
S: d5ecca53d38c4746be746b4be8a7904e81da6529 192.168.1.182:7002
  slots: (0 slots) slave
  replicates 4669f9ed7daa587c31de2dc2a0243be07049b8a6
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@node01 ~]#

2.12 免交互重新分片脚本

可以自动执行重新分片，而无需以交互方式手动输入参数。这可以使用如下命令行：

# 免交互重新分片脚本语法
redis-cli --cluster reshard <host>:<port> --cluster-from <node-id> --cluster-to <node-id> --cluster-slots <number of slots> --cluster-yes

# 将node01（192.168.1.181:7001）的1000个哈希槽分配给node02（192.168.1.182:7003）
[root@node01 ~]# redis-cli --cluster reshard 192.168.1.181:7000 --cluster-from 5277b27644e037f0d92e39a44dc6f40d7f422a3f --cluster-to 4669f9ed7daa587c31de2dc2a0243be07049b8a6 --cluster-slots 1000 --cluster-yes

2.13 自动重新分片

# 自动重新分片：
[root@node01 ~]# redis-cli --cluster rebalance 192.168.1.181:7000
>>> Performing Cluster Check (using node 192.168.1.181:7000)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Rebalancing across 3 nodes. Total weight = 3.00
Moving 999 slots from 192.168.1.182:7003 to 192.168.1.181:7001
#######################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################
Moving 1 slots from 192.168.1.182:7003 to 192.168.1.181:7000
#
[root@node01 ~]#

# 检查集群状态：发现分片已自动平衡
[root@node01 ~]# redis-cli --cluster check 192.168.1.181:7000
192.168.1.181:7000 (b7a5add9...) -> 0 keys | 5462 slots | 1 slaves.
192.168.1.181:7001 (5277b276...) -> 1 keys | 5461 slots | 1 slaves.
192.168.1.182:7003 (4669f9ed...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 1 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.181:7000)
M: b7a5add92ee8103a2032688c3a7728bc8db02b23 192.168.1.181:7000
  slots:[0-5460],[6460] (5462 slots) master
  1 additional replica(s)
M: 5277b27644e037f0d92e39a44dc6f40d7f422a3f 192.168.1.181:7001
  slots:[5461-6459],[6461-10922] (5461 slots) master
  1 additional replica(s)
S: 7b088ee93f02b3cc0b9018ffee0eaf7035328503 192.168.1.183:7004
  slots: (0 slots) slave
  replicates b7a5add92ee8103a2032688c3a7728bc8db02b23
S: aef97908c5b67a64f195f840e023e9187e19197a 192.168.1.183:7005
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
M: 4669f9ed7daa587c31de2dc2a0243be07049b8a6 192.168.1.182:7003
  slots:[10923-16383] (5461 slots) master
  1 additional replica(s)
S: d5ecca53d38c4746be746b4be8a7904e81da6529 192.168.1.182:7002
  slots: (0 slots) slave
  replicates 4669f9ed7daa587c31de2dc2a0243be07049b8a6
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@node01 ~]#

2.14 报错解决

2.14.1 创建实例报错

This systems seems to use systemd. Please take a look at the provided example service unit files in this directory, and adapt and install t hem. Sorry!

[root@node01 ~] vim ./install_server.sh
注释下面的代码即可
#bail if this system is managed by systemd
#_pid_1_exe="$(readlink -f proc/1/exe)"
#if [ "${_pid_1_exe##*/}" = systemd ]
#then
#       echo "This systems seems to use systemd."
#       echo "Please take a look at the provided example service unit files in this directory, and adapt and install them. Sorry!"
#       exit 1
#fi

2.14.2 在创建或者添加节点到集群时报错

[ERR] Node 192.168.1.181:7000 is not empty. Either the node already knows other nodes

(check with CLUSTER NODES) or contains some key in database 0.

# 在创建或者添加节点到集群时，报错：
[ERR] Node 192.168.1.181:7000 is not empty. Either the node already knows other nodes
(check with CLUSTER NODES) or contains some key in database 0.

# 将需要新增的节点下aof、rdb等本地备份文件及Redis.conf里面cluster-config-file所在的文件删除；
[root@node01 ~]# cd usr/local/redis/7000
[root@node01 7000]# ls
appendonly.aof dump.rdb nodes-7000.conf
[root@node01 7000]# rm -rf *

# 登录新节点对数据库进行清除：
[root@node01 7000]# redis-cli -h 192.168.1.181 -p 7000  
192.168.1.181:7000> flushdb
OK
192.168.1.181:7000>

# 重启新节点：
[root@node01 ~]# etc/init.d/redis_7000 restart

# 如果重新创建或者添加节点到时，依然报错，可以修复下集群：
[root@node01 ~]# redis-cli --cluster fix 192.168.1.181:7000
192.168.1.181:7000 (8fffaca4...) -> 0 keys | 0 slots | 0 slaves.
192.168.1.181:7001 (5277b276...) -> 0 keys | 10923 slots | 2 slaves.
192.168.1.182:7003 (4669f9ed...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.181:7000)
M: 8fffaca408475225eda34260a2793cdff0f7a5f2 192.168.1.181:7000
  slots: (0 slots) master
S: 7b088ee93f02b3cc0b9018ffee0eaf7035328503 192.168.1.183:7004
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
S: d5ecca53d38c4746be746b4be8a7904e81da6529 192.168.1.182:7002
  slots: (0 slots) slave
  replicates 4669f9ed7daa587c31de2dc2a0243be07049b8a6
S: aef97908c5b67a64f195f840e023e9187e19197a 192.168.1.183:7005
  slots: (0 slots) slave
  replicates 5277b27644e037f0d92e39a44dc6f40d7f422a3f
M: 5277b27644e037f0d92e39a44dc6f40d7f422a3f 192.168.1.181:7001
  slots:[0-10922] (10923 slots) master
  2 additional replica(s)
M: 4669f9ed7daa587c31de2dc2a0243be07049b8a6 192.168.1.182:7003
  slots:[10923-16383] (5461 slots) master
  1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@node01 ~]#

# 查询或者创建键值报错：
[root@node01 ~]# redis-cli -h 192.168.1.181 -p 7000
192.168.1.181:7000> get name
(error) MOVED 5798 192.168.1.181:7001
192.168.1.181:7000>

# 解决办法：启动时使用-c参数来启动集群模式，命令如下：
[root@node01 ~]# redis-cli -c -h 192.168.1.181 -p 7000
192.168.1.181:7000> get name
-> Redirected to slot [5798] located at 192.168.1.181:7001
"superman"
192.168.1.181:7001>

2.14.3 登陆或者创建集群报错

[ERR] Node 172.29.57.198:7000 NOAUTH Authentication required.

解决办法：
用redis-cli 密码登陆（redis-cli -a password）。

2.14.4 Redis服务停止报错

(error) NOAUTH Authentication required.

Redis服务器设置密码后，使用/etc/init.d/redis_7000 stop 会出现以下信息：

/etc/init.d/redis_7000
Stopping ...
OK
(error) NOAUTH Authentication required.
Waiting for Redis to shutdown ...
Waiting for Redis to shutdown ...
Waiting for Redis to shutdown ...
Waiting for Redis to shutdown ...
Waiting for Redis to shutdown ...
Waiting for Redis to shutdown ...
Waiting for Redis to shutdown ...
Waiting for Redis to shutdown ...

出现这样的错误信息，redis 这时是没有停止服务的。
可以使用ps -ef | grep redis 查进程号 然后kill 掉，如果在deamon下还需要去删除pid文件，有点繁琐。

解决办法：
用redis-cli 密码登陆（redis-cli -a password）。
再用ps -ef | grep redis 可以看到redis进程已经正常退出。

修改redis服务脚本，加入如下所示的红色授权信息即可：
vi etc/init.d/redis
$CLIEXEC -a "password" -p $REDISPORT shutdown

3 Redis Cluster集群相关操作

3.1 关闭集群的某个节点

比如关闭端口号为7000的实例

# redis-cli -a <password> -c -h 127.0.0.1 -p 7000 shutdown

[root@node01 ~]# redis-cli -a 123456 -c -h 10.123.188.134 -p 7000 shutdown
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
[root@node01 ~]#
[root@node01 ~]# redis-cli -a 123456 -h 10.123.188.134 -p 7001
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.1.181:7001> cluster nodes
2a1c901e41d20de14eeb62a3c3f70133584786ce 192.168.1.181:7000@17000 slave,fail d22374ac2f4fdc28db316b95a034176629a246bc 1615809932963 1615809930458 7 disconnected
d22374ac2f4fdc28db316b95a034176629a246bc 192.168.1.182:7003@17001 master - 0 1615810216120 7 connected 0-5460
c6df7738b24644317c629f669f63c7ca0d412fe1 192.168.1.183:7005@17001 slave 8237509c0dd48bea6a57859813aec8bb99e650d4 0 1615810217122 3 connected
97b6fc54a516f6ff20a3e6244d66dd2ea8dc139b 192.168.1.183:7004@17000 master - 0 1615810215118 5 connected 10923-16383
ccb235ca30cc8466459fdf5854aacf8653f7e760 192.168.1.181:7001@17001 myself,slave 97b6fc54a516f6ff20a3e6244d66dd2ea8dc139b 0 1615809928000 5 connected
8237509c0dd48bea6a57859813aec8bb99e650d4 192.168.1.182:7002@17000 master - 0 1615810216621 3 connected 5461-10922
192.168.1.181:7001>

关闭之后，可以将实例重新启动，启动完成之后，自动加入到集群当中；

192.168.1.181:7001> quit
[root@node01 ~]# systemctl restart Redis_7000
[root@node01 ~]# redis-cli -a 1qaz.@WSX.3edc -h 10.123.188.134 -p 7001
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.1.181:7001> cluster nodes
2a1c901e41d20de14eeb62a3c3f70133584786ce 192.168.1.181:7000@17000 slave d22374ac2f4fdc28db316b95a034176629a246bc 0 1615810611980 7 connected
d22374ac2f4fdc28db316b95a034176629a246bc 192.168.1.182:7003@17001 master - 0 1615810611579 7 connected 0-5460
c6df7738b24644317c629f669f63c7ca0d412fe1 192.168.1.183:7005@17001 slave 8237509c0dd48bea6a57859813aec8bb99e650d4 0 1615810610978 3 connected
97b6fc54a516f6ff20a3e6244d66dd2ea8dc139b 192.168.1.183:7004@17000 master - 0 1615810612080 5 connected 10923-16383
ccb235ca30cc8466459fdf5854aacf8653f7e760 192.168.1.181:7001@17001 myself,slave 97b6fc54a516f6ff20a3e6244d66dd2ea8dc139b 0 1615810610000 5 connected
8237509c0dd48bea6a57859813aec8bb99e650d4 192.168.1.182:7002@17000 master - 0 1615810611000 3 connected 5461-10922
192.168.1.181:7001>

3.2 集群的关闭

逐个关闭Redis实例即可。

[root@node01 ~]# systemctl stop redis_7000
[root@node01 ~]# systemctl stop redis_7001
[root@node02 ~]# systemctl stop redis_7002
[root@node02 ~]# systemctl stop redis_7003
[root@node03 ~]# systemctl stop redis_7004
[root@node03 ~]# systemctl stop redis_7005

3.3 Redis集群的重新启动

3.3.1 保留原来的数据

逐个关闭Redis实例，再逐个的启动即可。

3.3.2 丢弃原来的数据

关闭实例，清空实例中数据存放目录的所有内容，然后逐个启动实例，在任意一个实例上执行集群的创建命令即可,本质上就是创建一个新的集群。

关闭实例

[root@node01 ~]# systemctl stop redis_7000
[root@node01 ~]# systemctl stop redis_7001
[root@node02 ~]# systemctl stop redis_7002
[root@node02 ~]# systemctl stop redis_7003
[root@node03 ~]# systemctl stop redis_7004
[root@node03 ~]# systemctl stop redis_7005

[root@node01 ~]# rm -rf /usr/local/redis/7000/*
[root@node01 ~]# rm -rf /usr/local/redis/7001/*
[root@node02 ~]# rm -rf /usr/local/redis/7002/*
[root@node02 ~]# rm -rf /usr/local/redis/7003/*
[root@node03 ~]# rm -rf /usr/local/redis/7004/*
[root@node03 ~]# rm -rf /usr/local/redis/7005/*

启动实例

[root@node01 ~]# systemctl start redis_7000
[root@node01 ~]# systemctl start redis_7001
[root@node02 ~]# systemctl start redis_7002
[root@node02 ~]# systemctl start redis_7003
[root@node03 ~]# systemctl start redis_7004
[root@node03 ~]# systemctl start redis_7005

执行集群创建命令

[root@node01 ~]# /usr/local/redis/bin/redis-cli --cluster create 192.168.1.181:7000 192.168.1.181:7001 \
192.168.1.182:7002 192.168.1.182:7003 192.168.1.183:7004 192.168.1.183:7005 \
--cluster-replicas 1

3.4 Redis cluster命令手动管理Redis集群

1、打印集群的信息
CLUSTER INFO

2、列出集群当前已知的所有节点（node），以及这些节点的相关信息
CLUSTER NODES

3、将ip和port所指定的节点添加到集群中
CLUSTER MEET <ip> <port>

4、从集群中移除node_id指定的节点
CLUSTER FORGET <node_id>

5、将当前节点设置为node_id指定的节点的从节点
CLUSTER REPLICATE <node_id>

6、将节点的配置文件保存到硬盘里面
CLUSTER SAVECONFIG

7、将一个或多个槽（slot）指派（assign）给当前节点
CLUSTER ADDSLOTS <slot> [slot ...]

8、移除一个或多个槽对当前节点的指派
CLUSTER DELSLOTS <slot> [slot ...]

9、移除指派给当前节点的所有槽，让当前节点变成一个没有指派任何槽的节点
CLUSTER FLUSHSLOTS

10、将槽slot指派给node_id指定的节点，如果槽已经指派给另一个节点，那么先让另一个节点删除该槽>，然后再进行指派
CLUSTER SETSLOT <slot> NODE <node_id>

11、将本节点的槽slot迁移到node_id指定的节点中
CLUSTER SETSLOT <slot> MIGRATING <node_id>

12、从node_id指定的节点中导入槽 slot 到本节点
CLUSTER SETSLOT <slot> IMPORTING <node_id>

13、取消对槽slot的导入（import）或者迁移（migrate）
CLUSTER SETSLOT <slot> STABLE

14、计算键key应该被放置在哪个槽上
CLUSTER KEYSLOT <key>

15、返回槽slot目前包含的键值对数量
CLUSTER COUNTKEYSINSLOT <slot>

16、返回count个slot槽中的键
CLUSTER GETKEYSINSLOT <slot> <count>

4 Redis Cluster集群性能调优

4.1 关闭RDB防止fork进程的内存溢出问题

save ""
appendonly=yes

4.2 防止某个节点挂掉，整个cluster挂掉的问题

cluster-require-full-coverage no

4.3 设置最大内存

maxmemory

11453246122

4.4 日志

logfile "./redis.log"

4.5 防止因为磁盘同步，卡掉住进程的情况出现

no-appendfsync-on-rewrite yes

4.6 内核修改，如果没开防火墙可以不设置这个

net.nf_conntrack_max = 1648576

4.7 这个用来防止内存申请不到发生卡死的情况，很重要

vm.overcommit_memory = 1

有三种方式修改内核参数，但要有root权限：

编辑/etc/sysctl.conf ，改vm.overcommit_memory = 1，然后sysctl -p 使配置文件生效
sysctl vm.overcommit_memory = 1
echo 1 > /proc/sys/vm/overcommit_memory

建议使用方法1。

4.8 thp redis warning要求关掉

echo never > /sys/kernel/mm/transparent_hugepage/enabled

同时写入rc.local，保证下次重启生效

4.9 最大连接数要改一下，redis的warning会要求修改

net.core.somaxconn = 1024

编辑/etc/sysctl.conf ，增加 net.core.somaxconn= 1024，然后sysctl -p 使配置文件生效

4.10 设置内存超出策略

默认没有设置

allkeys-lru

BGREWRITEAOF 这个命令可以重写aof，因为aof长时间增量更新，导致越来越大，但是内存可能没这么大，所以可以用这个命令重写进行复制备份，恢复等
内存吃紧的时候可以尝试手动去掉内存碎片

hanease

分布式存储机制-槽

节点集群部署、节点的增加及删除和优化实战

1 Redis Cluster集群介绍

1.1 Redis集群的优势

1.2 Redis集群的工作原理

1.4 Redis集群的主从复制模型

2 Redis cluster部署

2.1 项目环境

2.2 Redis安装(6节点分别执行，以节点1为例)

2.3 创建Redis实例

2.3.1 创建节点1实例

2.3.2 创建节点2实例

2.3.3 创建节点3实例

2.3.4 创建节点4实例

2.3.5 创建节点5实例

2.3.6 创建节点6实例

2.4 开启Redis集群模式

2.5 重启Redis实例

2.6 创建集群

2.7 查看集群信息或状态

2.8 查看集群节点

2.9 删除节点

2.10 添加新的节点

2.11 重新分片

2.12 免交互重新分片脚本

2.13 自动重新分片

2.14 报错解决

2.14.1 创建实例报错

2.14.2 在创建或者添加节点到集群时报错

2.14.3 登陆或者创建集群报错

2.14.4 Redis服务停止报错

3 Redis Cluster集群相关操作

3.1 关闭集群的某个节点

3.2 集群的关闭

3.3 Redis集群的重新启动

3.3.1 保留原来的数据

3.3.2 丢弃原来的数据

3.4 Redis cluster命令手动管理Redis集群

4 Redis Cluster集群性能调优

4.1 关闭RDB防止fork进程的内存溢出问题

4.2 防止某个节点挂掉，整个cluster挂掉的问题

4.3 设置最大内存

4.4 日志

4.5 防止因为磁盘同步，卡掉住进程的情况出现

4.6 内核修改，如果没开防火墙可以不设置这个

4.7 这个用来防止内存申请不到发生卡死的情况，很重要

4.8 thp redis warning要求关掉

4.9 最大连接数要改一下，redis的warning会要求修改

4.10 设置内存超出策略

公告