redis集群在线迁移第二篇（redis迁移后调整主从关系，停掉14机器上的所有从节点）-实战二

变更需求为：

1、调整主从关系，所有节点都调整到10.129.51.30机器上
2、停掉10.128.51.14上的所有redis，14机器关机
  14机器下线迁移至新机房，这段时间将不能提供服务.

当前集群情况

1、一共是2台12个实例，6主6从，每台6个实例。
2、30上有3台master，没有分配槽位，需变更为从节点。
3、10.128.51.14 放在A机房，10.129.51.30放在B机房。
4、目前经过上一次的调整，把0-16383槽位从10.128.51.14都迁移到了10.129.51.30的新伪集群中。

//查看集群主从节点信息。进行分析

[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7736 //输出结果我进行了小范围的调整

10.129.51.30:7736> cluster nodes
//6台从,每2台从同步一个master,黄色代表对应。

4bee54a5049524a6280fdb8a8de00a5bb94ccaa1 10.128.51.14:7733@17733 slave d1c7d99e13a2d7317baf883dffa906470a606641 0 1646269992000 62 connected
07c99eae8176eb0e985c5a9914f42900d18fb39b 10.128.51.14:7736@17736 slave d1c7d99e13a2d7317baf883dffa906470a606641 0 1646269990000 62 connected
fb28e95eb37b8d827c0f48800c4d08e64c6fc335 10.128.51.14:7731@17731 slave d46f032ea50763de8353fd530535412df6ffdc00 0 1646269993529 60 connected
ff34a5fadf42078332445bcf41a2b59416fd92da 10.128.51.14:7734@17734 slave d46f032ea50763de8353fd530535412df6ffdc00 0 1646269992327 60 connected
41767372c36cc268872e86d96660a32bf540624f 10.128.51.14:7732@17732 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 0 1646269990324 63 connected
b9a55f739a65e6277e40cdbb806b5443c8aad66e 10.128.51.14:7735@17735 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 0 1646269994330 63 connected

//闲置
27f7e8dede5b68581486ce8cefdd656032baed70 10.129.51.30:7734@17734 master - 0 1646269994826 4 connected
32ee19af1f6a534c4014b9c41d387666f049354b 10.129.51.30:7735@17735 master - 0 1646269989315 0 connected
bc5051aef1a756c274907d23198fbb932c384151 10.129.51.30:7736@17736 myself,master - 0 1646269991000 6 connected
//带槽位的master
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 master - 0 1646269995327 60 connected 0-5460
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 master - 0 1646269993324 63 connected 5461-10922
d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 master - 0 1646269994000 62 connected 10923-16383

我们继续分析，我把主要信息调整了一下，删除掉了防碍分析的信息。只保留相互之间关系(黄色做为示例，标记了相互关系)

10.129.51.30:7736> cluster nodes
1、10.129.51.30:7731 主 0-5460, master标记 d46f032ea50763de8353fd530535412df6ffdc00 的两个从节点 
fb28e95eb37b8d827c0f48800c4d08e64c6fc335 10.128.51.14:7731  slave d46f032ea50763de8353fd530535412df6ffdc00 
ff34a5fadf42078332445bcf41a2b59416fd92da 10.128.51.14:7734  slave d46f032ea50763de8353fd530535412df6ffdc00  
 
2、10.129.51.30:7732 主 5461-10922, master标记 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 的两个从节点 
41767372c36cc268872e86d96660a32bf540624f 10.128.51.14:7732  slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4  
b9a55f739a65e6277e40cdbb806b5443c8aad66e 10.128.51.14:7735  slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 

3、10.129.51.30:7733 主 10923-16383, master标记 d1c7d99e13a2d7317baf883dffa906470a606641 的两个从节点
4bee54a5049524a6280fdb8a8de00a5bb94ccaa1 10.128.51.14:7733 slave d1c7d99e13a2d7317baf883dffa906470a606641 
07c99eae8176eb0e985c5a9914f42900d18fb39b 10.128.51.14:7736 slave d1c7d99e13a2d7317baf883dffa906470a606641 
//以上6台主机全部要关机

//master，这几台不变
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731 master  0-5460
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732 master  5461-10922
d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733 master  10923-16383

//需要等待调整为具有槽位主节点变成slave
27f7e8dede5b68581486ce8cefdd656032baed70 10.129.51.30:7734@17734 master - 
32ee19af1f6a534c4014b9c41d387666f049354b 10.129.51.30:7735@17735 master -  
bc5051aef1a756c274907d23198fbb932c384151 10.129.51.30:7736@17736 myself,master

根据第二次整理，我再次整理出相互关系表格，方便我下面实际操作。

主节点ID                                   主节点地址          从节点ID                                 从节点地址
//主槽位 0-5460
d46f032ea50763de8353fd530535412df6ffdc00  10.129.51.30:7731 fb28e95eb37b8d827c0f48800c4d08e64c6fc335 10.128.51.14:7731
d46f032ea50763de8353fd530535412df6ffdc00  10.129.51.30:7731 ff34a5fadf42078332445bcf41a2b59416fd92da 10.128.51.14:7734
//主槽位 5461-10922
91b7f8c79c91c7edd77458c332f0b9299bdb94d4  10.129.51.30:7732 41767372c36cc268872e86d96660a32bf540624f 10.128.51.14:7732
91b7f8c79c91c7edd77458c332f0b9299bdb94d4  10.129.51.30:7732 b9a55f739a65e6277e40cdbb806b5443c8aad66e 10.128.51.14:7735
//主槽位 10923-16383
d1c7d99e13a2d7317baf883dffa906470a606641  10.129.51.30:7733 4bee54a5049524a6280fdb8a8de00a5bb94ccaa1 10.128.51.14:7733
d1c7d99e13a2d7317baf883dffa906470a606641  10.129.51.30:7733 07c99eae8176eb0e985c5a9914f42900d18fb39b 10.128.51.14:7736

看到以上的信息，基本就已经很明确，redis集群中相互之间的关系，以及迁移动机和实际要达成的目标。

要达成目标为:

1、10.128.51.14:7731和10.128.51.14:7734 删除
2、10.129.51.30:7734从集群中移除 （原来不带槽位的master）
3、把10.129.51.30:7734节点，重新配置后，设置为当前集群中的10.129.51.30:7731主节点的从节点。

实际迁移、调整操作

一、设置新的主从关系（7734->7731）

1、删除从节点，在任意一台服务器上执行下面命令：
//操作1.删除从节点10.128.51.14:7734
[root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7731 ff34a5fadf42078332445bcf41a2b59416fd92da
>>> Removing node ff34a5fadf42078332445bcf41a2b59416fd92da from cluster 10.129.51.30:7731
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
//我们到另一台14机器上查下，已经被杀掉，也没有在cluster nodes里面出现.
//操作2.删除从节点 10.128.51.14:7731
[root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7731 fb28e95eb37b8d827c0f48800c4d08e64c6fc335
>>> Removing node fb28e95eb37b8d827c0f48800c4d08e64c6fc335 from cluster 10.129.51.30:7731
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
//我们看到10.128.51.14:7731从redis集群中被删除，并不会杀死该实例
解释:
a、删掉 10.129.51.30:7731 的两个从节点 10.128.51.14:7731 和 10.128.51.14:7734
b、解释 10.129.51.30:7731 为主节点地址， fb28e95eb37b8d827c0f48800c4d08e64c6fc335 为 10.128.51.14:7731 从节点标记
c、解释 10.129.51.30:7731 为主节点地址， ff34a5fadf42078332445bcf41a2b59416fd92da 为 10.128.51.14:7734 从节点标记
  从节点断开主从关系后并不会抛弃原有数据，只是无法再获取主节点上的数据变化:
2、把10.129.51.30:7734从集群中移除 （准备把该节点作为10.129.51.30：7731的从节点）
 语法: /redis-trib.rb del-node ip:端口号 节点id
[root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7734 d46f032ea50763de8353fd530535412df6ffdc00

[root@ht20 redis]# ps -ef | grep redis
root 40843 126564 0 12:34 pts/0 00:00:00 grep --color=auto redis
root 57063 1 0 Mar01 ? 00:03:53 /data/redis_fpmai1/redis/redis-server 10.129.51.30:7731 [cluster]
root 58394 1 0 Mar01 ? 00:03:25 /data/redis_fpmai2/redis/redis-server 10.129.51.30:7732 [cluster]
root 58881 1 0 Mar01 ? 00:07:27 /data/redis_fpmai3/redis/redis-server 10.129.51.30:7733 [cluster]
root 58893 1 0 Mar01 ? 00:03:03 /data/redis_fpmai4/redis/redis-server 10.129.51.30:7734 [cluster]
root 58939 1 0 Mar01 ? 00:02:54 /data/redis_fpmai6/redis/redis-server 10.129.51.30:7736 [cluster]
root 100693 1 0 Mar01 ? 00:02:45 /data/redis_fpmai5/redis/redis-server 10.129.51.30:7735 [cluster]
//我们看到10.128.51.14:7734 master从redis集群中被删除，并不会杀死该实例,不过还是要有几步处理。
//a、先停掉 10.129.51.30:7734 ,删除data里面的内容，否则添加主从关系会提示不成功node not empty。
[root@ht20 redis]# kill -9 58893 //7734进程id
//b、因为以前已经加入过集群，删除其中的信息
[root@ht20 redis]#rm -rf /data/redis_fpmai4/redis/data/*.*
//c、重新启动7734实例
[root@ht20 redis]# /data/redis_fpmai4/redis/redis-server /data/redis_fpmai4/redis/redis.conf

3、前面的准备工作做完之后，就可以把10.129.51.30：7731 mater上添加一个slave从节点（需要重新建立主从关系）
语法 ./redis-trib.rb add-node --slave --master-id 主节点id 从节点地址主节点地址

[root@ht20 redis]# ./redis-trib.rb add-node --slave --master-id d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7734 10.129.51.30:7731
>>> Adding node 10.129.51.30:7734 to cluster 10.129.51.30:7731
>>> Performing Cluster Check (using node 10.129.51.30:7731)
M: d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731
slots:0-5460 (5461 slots) master
0 additional replica(s)
M: d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733
slots:10923-16383 (5461 slots) master
2 additional replica(s)
S: 07c99eae8176eb0e985c5a9914f42900d18fb39b 10.128.51.14:7736
slots: (0 slots) slave
replicates d1c7d99e13a2d7317baf883dffa906470a606641
S: 41767372c36cc268872e86d96660a32bf540624f 10.128.51.14:7732
slots: (0 slots) slave
replicates 91b7f8c79c91c7edd77458c332f0b9299bdb94d4
M: 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732
slots:5461-10922 (5462 slots) master
2 additional replica(s)
M: 32ee19af1f6a534c4014b9c41d387666f049354b 10.129.51.30:7735
slots: (0 slots) master
0 additional replica(s)
M: bc5051aef1a756c274907d23198fbb932c384151 10.129.51.30:7736
slots: (0 slots) master
0 additional replica(s)
S: 4bee54a5049524a6280fdb8a8de00a5bb94ccaa1 10.128.51.14:7733
slots: (0 slots) slave
replicates d1c7d99e13a2d7317baf883dffa906470a606641
S: b9a55f739a65e6277e40cdbb806b5443c8aad66e 10.128.51.14:7735
slots: (0 slots) slave
replicates 91b7f8c79c91c7edd77458c332f0b9299bdb94d4
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.129.51.30:7734 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 10.129.51.30:7731.
[OK] New node added correctly.

//检查加入情况，14的2个节点已经删除，30的7734已变成slave

解释：
1、d46f032ea50763de8353fd530535412df6ffdc00 master标识
2、10.129.51.30:7734 要加入的新的机器 (之前加入过，所以需要停掉该实例，删除data目录下的内容，再重新启动该实例)
3、10.129.51.30:7731 是主节点,即有槽位的机器(当然也可以是其他任意主节点，有槽位的),

//检查下集群中是不是已经达到我们的目的(红色这一行就是我们的目标)。

[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7731
10.129.51.30:7731> cluster nodes
d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 master - 10923-16383
421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734@17734 slave d46f032ea50763de8353fd530535412df6ffdc00
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 myself,master - 0-5460
07c99eae8176eb0e985c5a9914f42900d18fb39b 10.128.51.14:7736@17736 slave d1c7d99e13a2d7317baf883dffa906470a606641
27f7e8dede5b68581486ce8cefdd656032baed70 :0@0 master,fail,noaddr //这一条就是设置主从时候，失误导致的
41767372c36cc268872e86d96660a32bf540624f 10.128.51.14:7732@17732 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 master - 5461-10922
32ee19af1f6a534c4014b9c41d387666f049354b 10.129.51.30:7735@17735 master - connected
bc5051aef1a756c274907d23198fbb932c384151 10.129.51.30:7736@17736 master - connected
4bee54a5049524a6280fdb8a8de00a5bb94ccaa1 10.128.51.14:7733@17733 slave d1c7d99e13a2d7317baf883dffa906470a606641
b9a55f739a65e6277e40cdbb806b5443c8aad66e 10.128.51.14:7735@17735 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4

所以看到红色的关键语句就4条。

回顾下主从的关系

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1、10.129.51.30:7731 主 0-5460, master标记 d46f032ea50763de8353fd530535412df6ffdc00 的两个从节点 
fb28e95eb37b8d827c0f48800c4d08e64c6fc335 10.128.51.14:7731  slave d46f032ea50763de8353fd530535412df6ffdc00 
ff34a5fadf42078332445bcf41a2b59416fd92da 10.128.51.14:7734  slave d46f032ea50763de8353fd530535412df6ffdc00  
 
2、10.129.51.30:7732 主 5461-10922, master标记 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 的两个从节点 
41767372c36cc268872e86d96660a32bf540624f 10.128.51.14:7732  slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4  
b9a55f739a65e6277e40cdbb806b5443c8aad66e 10.128.51.14:7735  slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 

3、10.129.51.30:7733 主 10923-16383, master标记 d1c7d99e13a2d7317baf883dffa906470a606641 的两个从节点
4bee54a5049524a6280fdb8a8de00a5bb94ccaa1 10.128.51.14:7733 slave d1c7d99e13a2d7317baf883dffa906470a606641 
07c99eae8176eb0e985c5a9914f42900d18fb39b 10.128.51.14:7736 slave d1c7d99e13a2d7317baf883dffa906470a606641 

//master
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 master - 0 1646269995327 60 connected 0-5460
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 master - 0 1646269993324 63 connected 5461-10922
d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 master - 0 1646269994000 62 connected 10923-16383

//等待调整为slave
27f7e8dede5b68581486ce8cefdd656032baed70 10.129.51.30:7734@17734 master - 0 1646269994826 4 connected
32ee19af1f6a534c4014b9c41d387666f049354b 10.129.51.30:7735@17735 master - 0 1646269989315 0 connected
bc5051aef1a756c274907d23198fbb932c384151 10.129.51.30:7736@17736 myself,master - 0 1646269991000 6 connected


  主节点ID                                 主节点地址      从节点ID                                从节点地址
//主槽位 0-5460
d46f032ea50763de8353fd530535412df6ffdc00  10.129.51.30:7731 fb28e95eb37b8d827c0f48800c4d08e64c6fc335 10.128.51.14:7731
d46f032ea50763de8353fd530535412df6ffdc00  10.129.51.30:7731 ff34a5fadf42078332445bcf41a2b59416fd92da 10.128.51.14:7734
//主槽位 5461-10922
91b7f8c79c91c7edd77458c332f0b9299bdb94d4  10.129.51.30:7732 41767372c36cc268872e86d96660a32bf540624f 10.128.51.14:7732
91b7f8c79c91c7edd77458c332f0b9299bdb94d4  10.129.51.30:7732 b9a55f739a65e6277e40cdbb806b5443c8aad66e 10.128.51.14:7735
//主槽位 10923-16383
d1c7d99e13a2d7317baf883dffa906470a606641  10.129.51.30:7733 4bee54a5049524a6280fdb8a8de00a5bb94ccaa1 10.128.51.14:7733
d1c7d99e13a2d7317baf883dffa906470a606641  10.129.51.30:7733 07c99eae8176eb0e985c5a9914f42900d18fb39b 10.128.51.14:7736

 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

二、设置新的主从关系（7735->7732）

二、 设置新的主从关系（7735->7732）
1、从redis集群中删掉两个从节点(10.128.51.14:7732 和10.128.51.14:7735)
     在任意一台服务器上执行下面命令：
     //删除10.129.51.30:7732的从节点10.128.51.14:7735和10.128.51.14:7732
     [root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7732 41767372c36cc268872e86d96660a32bf540624f
     [root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7732 b9a55f739a65e6277e40cdbb806b5443c8aad66e

2、移除7735节点
    [root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7735 32ee19af1f6a534c4014b9c41d387666f049354b
    [root@ht20 redis]# kill -9 58894  
    [root@ht20 redis]#rm -rf   /data/redis_fpmai5/redis/data/*.*
    [root@ht20 redis]# /data/redis_fpmai5/redis/redis-server /data/redis_fpmai5/redis/redis.conf
3、添加一个新的主从关系（重新建立主从关系）
 [root@ht20 redis]# ./redis-trib.rb add-node --slave --master-id 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7735 10.129.51.30:7732

三、设置新的主从关系（7736->7733）

三、 设置新的主从关系（7736->7733）
1、从redis集群中删掉两个从节点(10.128.51.14:7733 和10.128.51.14:7736)
     在任意一台服务器上执行下面命令：
     //删除10.129.51.30:7732的从节点10.128.51.14:7733 和10.128.51.14:7736
     [root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7733 4bee54a5049524a6280fdb8a8de00a5bb94ccaa1
     [root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7733 07c99eae8176eb0e985c5a9914f42900d18fb39b

2、移除7736节点
    [root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7736  bc5051aef1a756c274907d23198fbb932c384151
    [root@ht20 redis]# kill -9 58894  
    [root@ht20 redis]#rm -rf   /data/redis_fpmai5/redis/data/*.*
    [root@ht20 redis]# /data/redis_fpmai5/redis/redis-server /data/redis_fpmai5/redis/redis.conf
3、添加一个新的主从关系（重新建立主从关系）
 [root@ht20 redis]# ./redis-trib.rb add-node --slave --master-id d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7736 10.129.51.30:7733
  注意：del-node只能删除没有分配slot的节点

执行相关过程就单独放这里吧。这样可以和上面的操作分开，上面的二和三中的操作，看着基本很简洁了，

//删除7732节点
[root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7732 41767372c36cc268872e86d96660a32bf540624f
>>> Removing node 41767372c36cc268872e86d96660a32bf540624f from cluster 10.129.51.30:7732
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
//删除7732从节点
[root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7732 b9a55f739a65e6277e40cdbb806b5443c8aad66e
>>> Removing node b9a55f739a65e6277e40cdbb806b5443c8aad66e from cluster 10.129.51.30:7732
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

//删除集群中的节点（之前是标记为了master,但是槽位是空的）

[root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7735 32ee19af1f6a534c4014b9c41d387666f049354b
>>> Removing node 32ee19af1f6a534c4014b9c41d387666f049354b from cluster 10.129.51.30:7735
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

//删除集群中的节点（之前标记也是master,槽位为空）

[root@ht20 redis]# ./redis-trib.rb del-node 10.129.51.30:7736 bc5051aef1a756c274907d23198fbb932c384151
>>> Removing node bc5051aef1a756c274907d23198fbb932c384151 from cluster 10.129.51.30:7736
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

//为确保迁移过程中正确，检查一下当前集群情况

[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7731
10.129.51.30:7731> cluster nodes
d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733 master 10923-16383
421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734 slave d46f032ea50763de8353fd530535412df6ffdc00
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731 myself,master 0-5460
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732 master 5461-10922

//删除掉redis_fpmai5和redis_fpmai6的data目录下的文件，重新启动

[root@ht20 redis]# /data/redis_fpmai5/redis/redis-server /data/redis_fpmai5/redis/redis.conf
[root@ht20 redis]# /data/redis_fpmai6/redis/redis-server /data/redis_fpmai6/redis/redis.conf
[root@ht20 redis]# ps -ef | grep redis
root 42097 1 0 12:40 ? 00:00:07 /data/redis_fpmai4/redis/redis-server 10.129.51.30:7734 [cluster]
root 57063 1 0 Mar01 ? 00:04:03 /data/redis_fpmai1/redis/redis-server 10.129.51.30:7731 [cluster]
root 58394 1 0 Mar01 ? 00:03:35 /data/redis_fpmai2/redis/redis-server 10.129.51.30:7732 [cluster]
root 58881 1 0 Mar01 ? 00:07:40 /data/redis_fpmai3/redis/redis-server 10.129.51.30:7733 [cluster]
root 67736 1 0 14:35 ? 00:00:00 /data/redis_fpmai5/redis/redis-server 10.129.51.30:7735 [cluster]
root 67774 1 0 14:35 ? 00:00:00 /data/redis_fpmai6/redis/redis-server 10.129.51.30:7736 [cluster]

//执行中出现这样错误，我已经删除掉了data目录下的内容了，可能操作那块漏掉了，刚才操作有一步严谨。重新来一遍就不会出下面问题。

[root@ht20 redis]# ./redis-trib.rb add-node --slave --master-id 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7735 10.129.51.30:7732
>>> Adding node 10.129.51.30:7735 to cluster 10.129.51.30:7732
>>> Performing Cluster Check (using node 10.129.51.30:7732)
M: 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732
slots:5461-10922 (5462 slots) master
0 additional replica(s)
M: d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733
slots:10923-16383 (5461 slots) master
0 additional replica(s)
S: 421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734
slots: (0 slots) slave
replicates d46f032ea50763de8353fd530535412df6ffdc00
M: d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731
slots:0-5460 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[ERR] Node 10.129.51.30:7735 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

//我还是先杀掉7735,删除对应data目录下的文件，再重新启动7735实例，再次执行后正常

[root@ht20 redis]# /data/redis_fpmai5/redis/redis-server /data/redis_fpmai5/redis/redis.conf
[root@ht20 redis]# ./redis-trib.rb add-node --slave --master-id 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7735 10.129.51.30:7732
>>> Adding node 10.129.51.30:7735 to cluster 10.129.51.30:7732
>>> Performing Cluster Check (using node 10.129.51.30:7732)
M: 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732
slots:5461-10922 (5462 slots) master
0 additional replica(s)
M: d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733
slots:10923-16383 (5461 slots) master
0 additional replica(s)
S: 421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734
slots: (0 slots) slave
replicates d46f032ea50763de8353fd530535412df6ffdc00
M: d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731
slots:0-5460 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.129.51.30:7735 to make it join the cluster.
Waiting for the cluster to join..
>>> Configure node as replica of 10.129.51.30:7732.
[OK] New node added correctly.

执行到这里，我让开发人员测试了下目前redis的情况，暂时都一切正常。但是没有让k8s pod重新部署.

//最后执行一下

[root@ht20 redis]# ./redis-trib.rb add-node --slave --master-id d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7736 10.129.51.30:7733
>>> Adding node 10.129.51.30:7736 to cluster 10.129.51.30:7733
>>> Performing Cluster Check (using node 10.129.51.30:7733)
M: d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733
slots:10923-16383 (5461 slots) master
0 additional replica(s)
S: f0e5d3d17babbce85a9290e679a626bc0bbacc93 10.129.51.30:7735
slots: (0 slots) slave
replicates 91b7f8c79c91c7edd77458c332f0b9299bdb94d4
M: 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734
slots: (0 slots) slave
replicates d46f032ea50763de8353fd530535412df6ffdc00
M: d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731
slots:0-5460 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.129.51.30:7736 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 10.129.51.30:7733.
[OK] New node added correctly.

[root@ht20 redis]# ./redis-cli -c -h 10.129.51.30 -p 7731
10.129.51.30:7731> cluster nodes

//下面的关系，稍微整理了一下，如下:

d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 myself,master - 0 1646290141000 60 connected 0-5460
421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734@17734 slave d46f032ea50763de8353fd530535412df6ffdc00 0 1646290140700 60 connected

91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 master - 0 1646290140000 63 connected 5461-10922
f0e5d3d17babbce85a9290e679a626bc0bbacc93 10.129.51.30:7735@17735 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4 0 1646290141702 63 connected

d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 master - 0 1646290138698 62 connected 10923-16383
c3c7ba2d0709121e69c2881724a0c5be903a4a6a 10.129.51.30:7736@17736 slave d1c7d99e13a2d7317baf883dffa906470a606641 0 1646290142703 62 connected

迁移到此就基本结束了。这个阶段也没有出现任何问题。

为了稳妥，让开发人员进行重新的部署。来看下pod日志打印情况和业务使用情况

d1c7d99e13a2d7317baf883dffa906470a606641 10.129.51.30:7733@17733 master 10923-16383
421ccd9a53168359272b733869fcb8ad827aa655 10.129.51.30:7734@17734 slave d46f032ea50763de8353fd530535412df6ffdc00
d46f032ea50763de8353fd530535412df6ffdc00 10.129.51.30:7731@17731 myself,master - 0 1646290592000 60 connected 0-5460
27f7e8dede5b68581486ce8cefdd656032baed70 :0@0 master,fail,noaddr - 1646282374602 1646282369596 4 disconnected
91b7f8c79c91c7edd77458c332f0b9299bdb94d4 10.129.51.30:7732@17732 master - 0 1646290592000 63 connected 5461-10922
c3c7ba2d0709121e69c2881724a0c5be903a4a6a 10.129.51.30:7736@17736 slave d1c7d99e13a2d7317baf883dffa906470a606641
f0e5d3d17babbce85a9290e679a626bc0bbacc93 10.129.51.30:7735@17735 slave 91b7f8c79c91c7edd77458c332f0b9299bdb94d4

日志打印正常，使用都正常。